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PREFACE 


This  volume  is  a  part  of  the  evaluation  of  the  Professional 
Standards  Review  Organization  (PSRO)  program  conducted  during  fiscal 
years  1976  and  1977  by  the  Office  of  Planning,  Evaluation  &  Legislation 

c 

(OPEL)  of  the  Health  Services  Administration,  U.S.  Public  Health  Service, 
Department  of  Health,  Education,  and  Welfare.    The  emphasis  of  the 
evaluation  was  on  program  utilization  control,  quality  assurance  and 
cost-containment  consequences  and  program  costs.    The  evaluation  was 
conducted  at  a  time  when  the  PSRO  program  was  in  an  early  stage  of 
implementation.    It  indicates  what  impact  the  PSRO  program  had  achieved 
by  the  start  of  1977,  provides  baseline  data  for  future  evaluative  work, 
and  offers  tests  of  numerous  quality  assurance  and  utilization  control 
evaluation  methodologies.    Summarized  below  are  the  evaluation's  salient 
findings. 

The  evaluation's  primary  study  was  an  analysis  of  Medicare 
utilization  data  compiled  expressly  for  this  evaluation.    The  Medicare 
analysis  investigated  the  effects  of  the  PSRO  program  on  hospital  days 
of  care  per  1,000  enrollees,  admissions  per  1,000  enrollees  and  average 
length  of  stay.    Eighteen  active  PSRO  areas  were  compared  to  twenty-six 
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matched  non-active  PSRO  areas.    No  aggregate  PSRO  effect  was  found  on 
the  utilization  variables  studied.    This  implies  that  as  a  "treatment" 
the  PSRO  program  thus  far  has  not  impacted  on  Medicare  utilization  rela- 
tive to  other  forms  of  utilization  review  being  conducted  in  non-PSRO 
areas.    While  no  overall  utilization  savings  were  found,  individual 
experiences  varied.    Some  of  the  organizations  were  associated  with 
lower  (favorable)  utilization  while  others  reflected  higher  (unfavorable) 
utilization  than  their  matched  comparison  areas.    These  findings  imply 
that,  even  though  certain  PSROs  have  demonstrated  some  degree  of  cost- 
effectiveness,  the  PSRO  program  as  a  national  program  is  not  now  func- 
tioning as  an  effective  cost-containment  mechanism.    Findings  concerning 
appropriateness  of  care  are  inconclusive  since  both  increases  and 
decreases  in  utilization  are  consistent  with  heightened  appropriateness 
of  care. 

An  analysis  of  hospital  discharge  abstract  data,  primarily 
derived  from  the  Commission  on  Professional  and  Hospital  Activities 
(CPHA)  Professional  Activity  Study  (PAS)  system,  indicates  that  PSRO 
review  tends  to  be  associated  with  increases  in  case  mix  severity  for 
Medicaid  patients,  but  not  for  Medicare  patients.    The  same  data  indi- 
cate smaller  (unfavorable)  decreases  in  case  mix  adjusted  average  length 
of  stay  (ALOS)  for  Medicaid  patients  in  hospitals  under  PSRO  review 
than  in  non-PSRO  comparison  hospitals  and  greater  (favorable)  decreases 
in  ALOS  for  Medicare  patients.    It  was  also  found  that,  in  some  instances, 
the  proportion  of  longer  stays  in  PSRO  hospitals  experienced  a  relative 
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decline.    The  ALOS  findings,  both  favorable  and  unfavorable,  were 
statistically  significant  yet  rather  small  (on  the  order  of  2%).  The 
severity  finding,  while  tentative  for  numerous  methodological  reasons, 
does  suggest  that  PSROs  may  utilize  bed  days  more  efficiently. 

Quality- oriented  components  of  the  PSRO  program  are  just 
now  being  implemented.    A  study  of  a  sample  of  medical  care  evaluation 
studies  (MCEs)  indicates  that  these  studies  were  effective  in  identify- 
ing and  correcting  problems  of  medical  care  in  hospitals.    The  number 
of  initial  audits  reported  has  grown  to  over  a  thousand  per  quarter, 
but  the  number  of  reaudits  reported  is  small.   .Choice  of  MCE  topics 
is  likely  to  effectively  reveal  problem  areas  but  is  not  necessarily 
based  upon  a  critical  selection  process.    Profile  analyses,  which 
can  aid  in  the  focusing  of  both  concurrent  review  (CR)  and  MCE  studies, 
are  implemented  in  only  a  few  PSROs. 

It  was  found  that  the  estimated  cost  of  a  fully  implemented 
acute  care  hospital  PSRO  program  for  Federal  beneficiaries  is  $268.50 
million.    Associated  Federal  discharge  unit  costs  are  currently  $18.31 
for  primarily  delegated  PSRO  areas  and  $16.00  for  primarily  non-dele- 
gated PSRO  areas.    The  PSRO  program  is  more  expensive  than  its  predeces 
sors.    The  cost  of  the  original  utilization  review  (UR)  regulations  was 

found  to  be  $81.3  million  while  implementation  of  the  revised  UR  regula 
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tions  would  have  cost  about  $107.6  million.       In  order  to  "recover" 

V    Original  UR  cost  figure  adjusted  to  1976  cost  levels. 


iii 


its  costs  the  PSRO  program  would  have  to  reduce  utilization  rates 
of  covered  populations  by  1.6  to  2.05  percent. 

The  report  cautions  the  reader  against  reaching  final  conclu- 
sions about  the  program  since  the  evaluation  was  conducted  during  the 
time  the  program  was  in  the  developmental  stage.    However,  data  are 
sufficient  to  strongly  suggest  that  continued  PSRO  implementation  alone 
is  not  apt  to  cause  significant  changes  in  either  hospital  utilization 
rates  or  associated  governmental  expenditures. 

Considerably  more  study  is  urged  in  several  important  areas 
as  program  development  and  implmentation  is  continued.    Many  issues  have 
been  identified  in  the  course  of  the  evaluation  which  should  be  of  value 
in  deciding  on  the  direction  of  future  work. 
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1.  INTRODUCTION 

1 . 1  Objectives  of  the  Evaluation 

The  Medical  care  evaluation  (MCE)  study  is  the  primary  mechan- 
ism of  the  recently  established  PSRO  review  system  for  assuring  the 
quality  of  medical  care  delivered  in  institutions  throughout  the  United 
States.    Essentially,  the  MCE  is  an  audit  of  patient  care  in  terms  of 
its  efficiency,  quality  and  efficacy.    As  the  first  major  assessment  of 
nationwide  MCE  activity  scheduled  by  the  Bureau  of  Quality  Assurance, 
this  evaluation  deals  with  the  overall  level  and  distribution  of  MCEs 
performed  to  date,  the  extent  of  their  compliance  with  Bureau  policies 
and  perhaps  most  importantly,  their  effectiveness  in  identifying  and 
ultimately  removing  problem  areas  in  patient  care. 

MCE  audits  assure  or  improve  the  quality  of  health  care  through 
a  cyclical  procedure  based  on  a  system  of  peer  review.    The  MCE  cycle 
involves  those  discrete  operations  each  performed  locally  by  practitioners 
whose  own  performance  is  being  assessed.    First,  the  initial  audit,  a 
practice  which  has  been  promoted  for  a  number  of  years  by  organizations 
such  as  the  Joint  Commission  on  Accreditation  of  Hospitals  and  the 
American  Hospital  Association,  is  completed.    Secondly,  follow-up  pro- 
cedures consisting  of  appropriate  educational  or  administrative 
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interventions,  are  applied  where  deficiencies  in  medical  care  have 
been  found.    This  new  role  has  been  included  into  the  definition  of 
peer  review  within  the  past  four  years  by  the  Bureau  of  Quality  Assur- 
ance (BQA).    The  last  operation  of  the  cycle  is  the  reaudit,  which  serves  to 
demonstrate  improvement  in  the  deficient  situation  after  intervention. 
Thus,  MCEs  have  been  designed  to  ensure  that  problem  areas  are  not  only 
identified  but  are  also  removed.    This  evaluation  examines  MCEs  at  all 
three  stages  of  the  cyclical  process  so  as  to  closely  assess  their  total 
impact  on  medical  care  in  the  United  States  today  as  well  as  to  provide 
a  basis  for  projecting  their  potential  impact  in  the  future. 

This  evaluation  report  has  been  divided  into  three'major 
sections.    Section  1  is  a  status  report  on  the  overall  level  and  distri- 
bution of  MCE  activity  throughout  the  PSRO  system.    In  it,  both  the 
numbers  and  types  of  MCEs  performed  across  the  country  are  summarized. 
Section  2  examines  the  actual  building  blocks  of  the  MCE  program  by  look- 
ing in  detail  at  all  audit  and  reaudit  cycles  in  four  major  areas  of 
medicine  and  surgery.    These  studies  are  assessed  by  rating  systems  in 
terms  of  the  thoroughness  with  which  they  were  completed,  as  well  as  in 
terms  of  the  impact  they  succeeded  in  having  upon  medical  care.  Finally, 
Section  3  of  the  report  deals  with  the  entire  universe  of  MCEs  by  looking 
at  two  important  aspects  of  the  process,  topic  selection  and  follow-up 
procedures.    Both  of  these  areas  relate  very  directly  to  the  identifica- 
tion and  correction  of  problem  areas  in  medical  care  delivered  to  patients. 
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Only  if  topics  are  chosen  rigorously  is  it  possible  to  expect  that 
major  problem  areas  will  be  highlighted.    Similarly,  it  is  likely 
that  these  problems  will  be  corrected  only  if  follow-up  actions  are 
completely  carried  out.    Based  on  the  analysis  of  topic  selection  and 
follow-up,  some  conclusions  can  be  drawn  as  to  whether  the  MCE  program 
actually  has  the  potential  to  produce  results. 

Despite  the  fact  that  the  first  audits  were  completed  in  the 
early  1970s,  this  is  the  first  major  evaluation  of  their  quality  and 
potential  impact.    One  of  the  evaluation's  significant  products  has  been 
the  development  of  techniques  for  the  assessment  of  MCEs  now  as  well  as 
in  the  future.    Since  the  data  used  in  this  evaluation  are  limited  to  only 
two  years  of  activity,  and  since  that  activity  was  concentrated  towards 
the  end  of  the  time  period,  portions  of  the  evaluation  may  have  been  per- 
formed prematurely.    Nevertheless,  methodology  now  exists  for  continuing 
the  assessment  of  MCEs.    The  Health  Standards  and  Quality  Bureau  (HSQB) 
as  successor  to  BQA  will  continue  the  role  of  evaluator  for  subsequent 
examinations  of  the  program  by  reapplying  these  methodologies  to  later 
samples  of  MCEs  in  1978.    The  primary  thrust  of  future  evaluations  will 
be  to  measure  directly  the  impact  of  medical  audit  on  patient  care  and 
outcome.    Some  considerations  relevant  to  implementing  a  long-term  evalua- 
tion of  the  MCE  audit/reaudit  cycle  have  been  discussed  in  the  last 
chapter  of  this  report. 


-3- 


1 • 2  Summary  of  Evaluation  Findings 

An  intent  of  the  PSRO  legislation  is  to  ensure  that  health  care 
is  provided  to  the  recipients  of  Title  V,  Title  XVIII,  and  Title  XIX 
program  services  at  an  acceptable  level  of  quality.    The  aspect  of  the 
PSRO  program  which  especially  concentrates  on  the  quality  of  care  is  the 
Medical  Care  Evaluation  (MCE)  study  component.    This  subsection  summarizes 
report  findings  on  MCEs. 

Medical  audits,  the  primary  part  of  the  MCE  process,  are  not 
unique  to  the  PSRO  program.    They  have  a  precedent  in  hospital  activities 
performed  in  response  to  requirements  of  the  Joint  Commission  on  Accredita- 
tion of  Hospitals  (JCAH).    They  were  also  mandated  by  a  section  of  the 
original  (1966)  Medicare  regulations.    The  significant  contributions  of 
the  PSRO  program  have  been  to  make  the  PSRO  responsible  for  seeing  that 
these  evaluations  are  carried  out  and  more  importantly  to  require  that 
the  effectiveness  of  action  to  correct  deficiencies  in  care  found  in  the 
initial  audit  be  confirmed  by  a  follow-up  evaluation,  or  reaudit. 

Most  PSROs  have  implemented  MCEs  after  the  time  work  was  started 
on  concurrent  review.    In  general,  this  has  been  about  one  year  after  the 
PSRO  achieved  conditional  status.    When  the  PSRO  evaluation  was  in  its 
formative  stages  little  information  about  MCE  activities  was  available. 
As  the  study  progressed,  however,  it  became  evident  that  reliable  MCE 
data  were  becoming  more  abundant,  an-d  a  joint  effort  was  begun  by  the 
Health  Standards  and  Quality  Bureau,  HCFA  (previously  the  Bureau  of 
Quality  Assurance)  and  OPEL  to  evaluate  the  program's  MCE  activities. 
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The  examination  of  MCE  study  activities  was  based  on  three 
strategies:  (a)  to  provide  a  status  report  on  the  overall  level  and 
distribution  of  MCE  activity,  (b)  to  determine  the  effectiveness  of 
the  MCE  process—defined  as  changes  of  significance  in  medical  care 
process  and/or  patient  outcomes—by  examining  the  results  of  a  sample  - 
of  completed  MCEs  in  four  topic  categories  judged  to  have  high  impact 
potential,  and  (c)  to  assess  national  MCE  activity  as  a  whole  by 
examining  topic  selection  and  follow-up  procedures  among  the  universe 
of  studies  reported  as  of  June  1,  1977. 

Evaluation  was  made  possible  by  the  existence  of  the  PSRO 
Management  Information  System  (PMIS),  through  which  reports  of  com- 
pleted audits  and  reaudits  become  available  for  analysis.    Lags  in  the 
data  processing  system,  however,  can  result  in  a  delay  of  up  to  seven 
months  from  the  time  a  MCE  is  completed  to  the  time  it  is  included  in 

the  data  base.    Thus,  the  number  of  MCEs  reported,  especially  for  the 
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more  recent  quarters,  tends  to  be  underestimated.       However,  for  the 
purpose  of  this  review,  it  has  been  assumed  that  the  studies  reported 
in  the  data  base  fairly  represent  the  areas  of  medical  care  in  which 
studies  have  been  done. 

MCE  Status  Report 

Data  available  through  the  PMIS  show  that  PSRO  medical  care 
evaluation  activity  is  rapidly  accelerating  across  the  country.  While 

T7    The  data  base  available  for  these  analyses  as  of  June  1,  1977, 
for  example,  included  367  initial  audits  reported  for  1975, 
2,517  for  1976  and  20  for  the  first  quarter  of  1977.    As  of 
August  15,  1977,  these  figures  were  392,  3,467,  and  1,342 
respectively. 
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only  a  few  hundred  audits  were  reported  for  1975,  several  thousand 
MCE  studies  were  reported  for  1976,  and  the  quarterly  figures  for 
1977  are  still  rising.    Progress  in  MCE  implementation  is  not  uniform, 
however.    Of  the  56  PSROs  reporting  MCEs  in  calendar  years  1975  and 
1976,  ten  accounted  for  45%  of  the  total  studies.    Slightly  more  than 
45%  of  the  audits  were  done  on  medical  topics  and  about  the  same  pro- 
portion on  surgical  topics,  with  the  remaining  9%  focusing  on  adminis- 
trative and  miscellaneous  procedures.    When  the  audit  topics  were 
classified  into  305  topic  categories,  the  20  most  frequent  topic 
categories  accounted  for  over  half  of  the  studies  done.  Myocardial 
infarction,  pneumonia,  cholecystectomy,  and  appendectomy  were  the  four 
most  frequently  audited  topic  categories.    The  data  suggest  that  the 
predominance  of  the  top  twenty  topic  categories  is  lessening  over  time, 
but  tests  of  rank  correlation  show  that  the  trend  is  not  statistically  ' 
significant  thus  far. 

No  attempt  was  made  to  evaluate  the  compliance  of  hospitals 
or  PSROs  with  requirements  that  a  minimum  number  of  initial  MCEs  be 
submitted  each  year.    Since  this  requirement  was  issued  in  January  1977, 
compliance  cannot  be  assessed  before  March  1978. 

Assessment  of  a  Sample  of  Completed  MCEs 

The  second  protocol  of  the  MCE  eval uaticn' examined  studies 
which  had  completed  the  audit/reaudit  cycle.    Due  to  lags  in  the  report- 
ing system  and  the  fact  that  reaudits  need  not  be  completed  until  a 
year  after  the  initial  audit,  the  number  of  such  studies  represented 


in  the  PMIS  data  base  is  small.    Of  the  132  completed  studies  reported 
in  the  PMIS  and  others  solicited  in  a  canvas  of  the  PSROs,  37  in  the 
four  most  frequently  reported  topic  categories  were  selected  for  analysis. 
Documentation  for  these  studies  was  solicited  from  the  hospitals  through 
the  PSROs. 

The  evaluation  questions  asked  of  these  studies  were:  Was 
the  study  process  done  well?   Were  deficiencies  in  medical  care  identi- 
fied?   Was  action  taken  to  correct  these  deficiencies?    Was  improvement 
in  care  demonstrated  in  terms  of  reduced  numbers  of  deficiencies  found 
at  the  time  of  the  reaudit? 

Taken  in  total,  most  (95%)  of  the  MCE  studies  examined  were 
judged  to  have  followed  acceptable  research  procedures.    Whether  or  not 
such  procedures  were  acceptable,  each  of  the  MCE  studies  examined  identi- 
fied significant  deficiencies  and  most  reduced  the  rate  of  variation  from 
criteria  by  the  time  of  reaudit.    The  deficiencies  seem  to  fall  into  six 
major  problem  areas.    These  include  failure  to  complete  routine  admission 
procedures  and  work-up,  failure  to  follow  accepted  diagnostic  procedures, 
failure  to  provide  adequate  disease-specific  treatment,  inappropriate 
utilization,  unjustified  patient  outcomes,  and  failure  to  adequately 
document  patient  progress. 

When  analysis  is  performed  on  complete  audit  and  reaudit  .combina- 
tions, the  MCE  appears  to  be  an  effective  tool  for  the  discovery  and 
resolution  of  problems.  Good  results  were  produced  from  most  medical  audits 
whether  or  not  their  process  was  optimal.    Problem  areas  were  identified 
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in  both  the  process  of  care  and  patient  outcome.    A  significant  finding 
of  the  study  is  that  among  the  MCEs  in  this  sample,  68%  of  all  deficiencies 
found  after  initial  audits  were  removed  by  the  time  of  reaudit. 

Assessment  of  MCE  Topic  Selection  and  Fol low-Up 
Although  overall  impact  assessment  of  MCEs  is  premature  due 
to  the  small  number  of  reaudits  reported  at  this  time,  two  necessary 
conditions  for  overall  program  impact  can,  however,  be  examined  among 
the  vast  majority  of  studies  which  have  been  reported  only  through  the 
initial  audit:  whether  MCE  topics  being  chosen chave  the  potential  to 
identify  important  problem  areas  and  whether  action  is  being  taken  to 
remove  problems  once  they  are  identified. 

Present  PSRO  practices  in  choosing  topics  are  consistent  with 
the  goal  of  identifying  problem  areas  in  medical  care  and  patient  out- 
come.   The  degree  of  implementation  of  follow-up  procedures  also  appears 
at  this  time  to  be  consistent  with  the  goal  of  removing  these  problem 
areas.    In  selecting  medical  topics,  PSROs  tend  to  choose  diseases  which 
are  the  chief  causes  of  death  in  the  country,  except  for  carcinoma.  The 
surgical  procedures  which  are  most  frequently  carried  out  in  this 
country,  except  those  relating  to  trauma,  tend  to  be  the  most  frequently 
selected  surgical  topic  categories.    By  choosing  these  frequent  diseases 
and  procedures,  which  encompass  many  patients,-  physicians,  and  ancillary 
services,  the  program  has  a  reasonable  chance  of  effectively  uncovering 
problem  areas  at  this  time. 
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Aside  from  the  fact  that  topics  are  chosen  because  they  are 
nationally  common  reasons  for  hospital  admission,  there  seems  to  be 
no  other  strong  rationale  at  this  point  for  choosing  MCE  topics.  There 
is  insufficient  evidence  that  topics  are  chosen  to  address  perceived 
problems  in  care  at  the  hospital  level,  or  that  the  topics  selected  on 
the  whole  are  adequately  focused.    There  appears  to  be  a  bias  in  favor 
of  choosing  acute,  rather  than  chronic  conditions  for  examination.  While 
the  present  PSRO  practices  for  topic  selection  are  consistent  with  the 
discovery  of  problem  areas,  the  practices  will  have  to  be  refined  in 
the  future  to  permit  the  continued  selection  of  high-impact  topics  and 
to  ensure  that  biases  in  the  selection  process  do  not  leave  areas  of 
medical  care  unexamined.    Such  refinement  would  include:    (a)  greater 
emphasis  on  immediate,  perceived  problems  in  care  in  selection  of  topics 
and  (b)  making  topics  chosen  more  focused  and  specific. 

Based  on  the  analysis  of  2,876  audits,  deficiencies  in  the 
process  of  care  are  found  in  almost  70%  of  MCEs.    The  data  indicate 
that  almost  all  of  the  audits  which  detect  such  problems  recommend 
corrective  action.    A  review  of  a  10%  random  sample  of  studies  begun 
in  1975  shows  that  most  recommendations  are  implemented.    Thus,  though 
the  program  has  no  routine  way  of  monitoring  the  implementation  of  correc- 
tive action,  the  fragmentary  information  available  indicates  that  the 
frequency  of  such  action  is  consistent  with  the  widespread  removal  of 
problem  areas  in  medical  care.    Not  all  initial  audits  which  reveal 
deficiencies  in  care  call  for  reaudits;  in  this  sample  only  43.5%  of  all 
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initial  audits  required  them.    Only  132  of  these  reaudits  have,  how- 
ever, been  reported  as  complete.    Thus,  PSROs  are  not  getting  feed- 
back on  the  work  of  their  recommended  actions.    In  order  that  the 
changes  brought  about  in  the  quality  of  care  can  be  assessed,  reaudits 
will  have  to  be  carried  out  with  greater  regularity. 

Profile  Analysis  and  MCEs 

The  analysis  of  computer  profiles  which  aggregate  unit-record 
patient  data  in  order  that  patterns  of  care  can  be  observed  may  become 
a  powerful  tool  for  the  improvement  of  the  health  care  system.  As 
envisioned  in  the  design  of  the  PSRO  program,  profile  analysis  is  a  key 
component,  guiding  both  concurrent  review  and  medical  care  evaluation 
by  pointing  out  problems  of  high  priority.    As  the  PSRO  program  has  been 
implemented,  however,  profile  analysis  has  been  the  last  component  to  be 
put  in  place.    The  implementation  of  profile  analysis  is  a  complex  task. 
Not  only  must  a  profile  system  be  designed,  but  arrangements  must  be 
made  to  carry  out  the  practical  details  of  abstracting  and  processing 
the  data.    To  date,  few  PSROs  have  operational  profiling  systems. 

Profiling  can  be  especially  significant  to  the  development  of 
the  MCE  component  of  the  program.    Properly  applied,  profile  analysis 
can  provide  assistance  to  PSRO  physicians  by  objectively  identifying 
problems  in  their  own  hospitals  and  problems  on  a  larger  scale  which 
might  otherwise  be  unapparent.    Where  such  systems  exist  they  have 
generated  topics  for  both  areawide  and  individual  hospital  MCE  studies. 
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PSROs  and  the  Quality  of  Care 

This  evaluation  has  established  that  the  MCE  process,  as  imple- 
mented by  PSROs  and  delegated  hospitals,  is  an  effective  tool  to  detect 
and  remedy  problems  in  medical  care  in  particular  settings.    It  is  another 
question,  and  one  which -this  evaluation  has  only  begun  to  address,  whether 
the  MCE  program  effectively  fulfills  the  legal  mandate  to  maintain  the 
quality  of  the  care  provided  to  Federal  beneficiaries.    The  study  found 
that  the  growth  in  MCE  implementation  and  the  types  of  areas  of  medical 
care  selected  for  study  are  both  consonant  with  this  goal,  but  the 
present  study  is  not  adequate  to  assess  the  quality  of  health  care  on 
a  national  scale.    To  do  so  will  require  the  development  of  measures 
and  an  appropriate  experimental  design. 

1.3  Organization  of  the  Report 

This  report  is  divided  into  six  chapters.    The  first  two 
chapters  provide  introductory  material  as  well  as  a  short  history  of  the 
medical  audit  emphasizing  the  development  of  the  MCE  component  of  the 
PSRO  program  and  its  current  status.    The  following  three  chapters 
constitute  the  main  body  of  the  report,  and  reflect  the  analysis  and 
summary  of  all  the  data  collected  and  examined  during  the  evaluation 
procedure.    These  three  chapters  and,  hence,  the  evaluation  plan  itself 
are  outlined  below.    The  final  chapter  discusses  considerations  for 
further  MCE  evaluations. 
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OUTLINE  OF  EVALUATION  PLAN 


(CHAPTER  3)  ACTIVITY  IN  MCE  PROGRAM  BY: 

A.  Numbers  of  Audits 

B.  Categories  of  Topics 

C.  Types  of  Categories 

(medical,  surgical  and  administrative) 

D.  Distribution  of  Activity  . 

E.  Compliance 

F.  Conclusions  as  to: 

1)  level  and  type  of  activity 

2)  level  of  compl iance 

(CHAPTER  4)  EVALUATION  OF  A  SAMPLE  OF  MCEs  IN  FOUR 
MAJOR  TOPIC  CATEGORIES: 

A.  Rating  of  Individual  MCE  Studies  According  to: 

1)  Process 

2)  Impact 

B.  Assessment  of  Each  of  the  Four  Categories  in  Terms  of: 

1)  The  type  of  deficiencies  found 

2)  The  success  of  MCEs  in  that  category  in 
diminishing  or  eliminating  problem  areas 

(CHAPTER  5)  RESULTS  EXPECTED  IN  THE  WHOLE  UNIVERSE 
OF  MCE  ACTIVITY  ON  THE  BASIS  OF: 

A.  Quality  of  Topic  Selection  with  the  Following  Criteria 

1)  Scope  of  topic  category 

2)  Topic  specificity 

3)  Situation  specific  needs 

B.  Follow-up 

1)  Implementation  of  recommended  actions 

2)  Performance  of  reaudits 
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2.1  THE  DEVELOPMENT  OF  THE  MEDICAL  AUDIT 

The  medical  care  evaluation  component  of  the  PSRO  program  is  not  a 
new  invention,  but  rather  is  a  product  of  the  evolution  in  this  century  of 
both  the  technique  of  the  medical  audit  and  of  professional  and  social  concern 
regarding  the  need  to  systematically  monitor  and  improve  the  quality  of 
hospital  care.    This  section  will  review  these  parallel  developments. 

Bringing  American  Medicine  into  the  Twentieth  Century. 

It  is  difficult  to  comprehend  the  recency  of  the  changes  which 

have  brought  the  American  health  care  system  to  its  present  state  of 

technological  and  organizational  sophistication.    Lembcke  reminds  us 

of  the  situation  of  medical  education  in  the  era  immediately  preceding 

Abraham  Flexner's  renowned  report  to  the  Carnegie  Foundation: 

In  the  period  between  1900  and  1909,  some  165 
medical  schools  had  been  turning  out  5,200  doctors 
annually,  a  majority  of  whom  had  not  even  a  high 
school  education  before  taking  24  to  30  months 
of  medical  "education"  that  often  consisted  largely 
of  lectures  and  cramming  for  examinations,  with 
little  or  no  laboratory  and  clinical  work  worthy 
of  the  name. (1 ) 

Flexner's  tudy  of  medical  education,  published  in  j 
1910,  was  a  symptom  of  the  concern  of  the  leaders  of  the 


medical  profession  about  the  quality  of  medical  care.    The  Flexner  report 
also  was  a  stimulus  to  the  medical  profession  to  improve  the  quality  of 
care.    The  movement  to  reform  medical  education  was  joined  during  the  decade 
of  1910-1920  by  a  drive  to  raise  the  standards  of  hospital  care,  particularly 
on  the  part  of  the  Clinical  Congress  of  Surgeons  of  North  America  (soon 
to  be  succeeded  by  the  American  College  of  Surgeons). 

The  Clinical  Congress  of  Surgeons  advocated  a  nationwide  sur- 
vey of  hospitals  along  the  lines  of  Flexner' s  study,  judging  the 
performance  of  each  hospital  individually  according  to  objective  criteria 
of  performance,  based  on  a  systematic  analysis  of  case  records.  In 
this  they  were  following  the  example  of  a  key  figure,  Dr.  Emory  Codman, 
who  had  founded  his  own  hospital  and  in  it  instituted  the  pioneer  "end 
result"  medical  audit  system: 

Codman  described  his  forms  and  method;  and  also 
published  an  abstract  of  every  case  admitted  to 
his  hospital  in  the  years  1912  through  1916,  clas- 
sifying the  results  as  favorable  or  unfavorable, 
and  assigning  responsibility  for  the  latter  to 
errors  in  diagnostic  and  technical  ability,  poor 
surgical  judgment,  inadequate  care  or  equipment, 
and  the  type  of  disease  or  the  patient's  refusal 
of  treatment.    Codman' s  system  required  also  at 
least  one  revaluation  of  the  case  a  year  or  more 
later,  until  a  condition  of  stability  permitting 
final  judgment  had  been  reached. (2) 

Codman' s  system  was  not  adopted  by  any  other  hospitals  at  the 
time,  and  indeed  he  himself  had  to  abandon  it  when  he  rejoined  Massachu- 
setts General  Hospital.    Yet  his  position  as  head  of  the  Committee  on 
Hospital  Standardization  of  the  CCSNA  was  influential  in  persuading  the 
Carnegie  Foundation  in  1916  to  sponsor  a  survey  of  hospitals  with  over 


100  beds  in  the  United  States  and  Canada.    Under  the  direction  of 
Dr.  J.  G.  Bowman  of  the  American  College  of  Surgeons,  the  study 
applied  objective  criteria  to  salient  points  of  diagnosis,  treat- 
ment, and  outcomes  in  692  hospitals.    The  criteria  employed  and 
the  findings  in  the  hospitals  surveyed  were  never  published  nor 
have  they  been  preserved.    It  is  known  that  only  89  of  the  692 
hospitals  could  meet  any  reasonable  standards,  and  it  is  likely  that 
the  findings  were  so  shocking  that  the  survey  committee  ordered  the 
individual  survey  reports  destroyed  forthwith.  (3,4) 


The  Retreat  to  Structural  Criteria  of  Care  and  the 
Founding  of  the  JCAH 

The  American  College  of  Surgeons  continued  its  Hospital 
Standardization  program  from  1918  to  1952,  but  replaced  its  concern 
with  the  process  and  outcome  of  care  with  minimal  structural  standards 
for  hospital  organization,  staffing,  facilities,  procedures,  and  grad- 
uate training.    There  can  be  no  doubt  that  this  long-range  program  to 
improve  quality  of  care  by  less  direct  methods  had  a  salutory  effect, 
but  as  Lembcke  points  out,  adequacy  of  structure  does  not  guarantee 
quality  of  care: 

There  can  be  no  reasonable  doubt  that  a  hospital 
complying  in  spirit  as  well  as  form  with  the 
principles  of  "hospital  standardization"  .  .  . 
will  furnish  medical  care  of  good  quality.  Faith- 
fully construed,  the  principles  required  a  staff 
of  thoroughly  trained  physicians,  supplied  with 
adequate  facilities,  working  within  the  limits  of 
their  professional  competence,  following  generally 
accepted  procedures,  being  advised  by  their 
colleagues  in  serious  or  doubtful  cases,  and  under- 
going professional  scrutiny  of  their  performance 


through  regular  review  of  the  medical  records  of 
their  cases.    Yet  it  is  possible  for  such  generally 
phrased  standards  to  be  met  in  a  superficial  manner 
that  may  gain  "approval"  but  does  not  necessarily 
guarantee  medical  care  of  good  quality.    This  is 
so  because  compliance  may  exist  in  widely  varying 
degrees,  and  because  such  standards  as  those  con- 
cerning staff  organization  or  completeness  of  medi- 
cal records  are  not  related  directly  to  patient  care 
as  the  clinician  sees  it,  but  even  more  so  because 
the  criteria  for  performance  are  not  explicit.  (5) 

The  passage  of  the  Hospital  Survey  and  Construction  Act  of  1946 
(Hill-Burton)  was  in  some  ways  the  culmination  of  the  decades  of  Hospital 
Standardization  review,  for  the  law  required  that  every  state  receiving 
subsidies  have  a  hospital  licensure  law.    At  that  time,  only  ten  of  the  forty 
eight  states  had  such  laws.    Many  of  the  ACS  criteria  were  embodied  in  the 
federal  model  statute  and  crystallized  into  state  law. 

In  1952,  a  wider  organizational  base  for  accreditation  and 
review  was  achieved  through  the  formation  of  the  Joint  Commission  for 
Accreditation  of  Hospitals  (JCAH)  by  the  American  College  of  Surgeons, 
the  American  College  of  Physicians,  the  American  Medical  Association, 
and  the  American  Hospital  Association.    The  Joint  Commission  not  only 
had  a  wider  base  of  legitimation  for  its  criteria  and  standards,  but  also 
a  larger  field  staff.    In  the  beginning  it  adopted  the  same  standards  as 
the  ACS,  but  has  extended  them  over  the  twenty-five  years  of  its  influence. 

One  significant  addition  was  the  specification  of  what  Roemer 
and  Friedman  call  "control  committees"  in  the  medical  staff.    The  ACS 
standards  had  stipulated  that  "the  medical  staff  review  and  analyze  at 
regular  intervals  their  clinical  experience"  on  the  basis  of  patient 


records,  but  had  specified  no  mechanisms  for  carrying  out  such  review. 
The  JCAH  established  as  a  requirement  for  accreditation  that  there  be 
a  medical  records  committee  and  a  tissue  committee.  These  requirements 
have  since  expanded  to  include  infections,  pharmacy  and  utilization  review 
committees,  among  others. (6)    Lembcke  maintains,  further,  that  the  prospect 
of  a  thorough  external  audit  of  care  has  tended  to  encourage  hospitals  to  use 
these  control  committees  as  effective  internal  auditors. (7) 

i 

Federal  Involvement.    Until  the  passage  of  the  Medicare  and 
Medicaid  Amendments  of  1965,  the  story  of  peer  review  was  for  the  most 
part  one  of  voluntary  self-regulation  of  professionals  and  hospitals, 
through  their  own  associations.    With  the  advent  of  Federal  funding, 
the  interests  of  the  government  and  the  taxpayers,  as  well  as  the 
recipients  of  care,  in  the  efficiency  and  efficacy  of  the  care  provided 
have  become  bases  for  government  regulation.    Without  question,  the 
professions  retain  major  influence  on  the  content  of  legislation  and 
regulations,  but  the  impact  of  the  concerns  of  the  public  and  its 
representatives  with  cost  and  quality  can  easily  be  seen  in  such  forums 
as  the  Senate  Finance  Committee  hearings  which  preceded  the  passage 
of  the  PSRO  legislation. (8) 

The  Conditions  for  Participation  of  Hospitals  pursuant  to  the 
1965  Medicare  law  (P.L.  89-97)  specified  more  stringent  standards  than 
were  at  that  time  required  by  the  JCAH. (9)    The  regulations  also  contained 
language  which  mandated,  but  did  not  specifically  require,  medical  audits: 
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(f)    Standard;  Reviews.  -  (1)    Reviews  are  made,  on 
a  sample  or  other  basis,  of  admissions,  duration  of 
stays,  and  professional  services  furnished,  with 
respect  to  the  medical  necessity  of  the  services, 
and  for  the  purpose  of  promoting  the  most  efficient 
use  of  available  health  facilities  and  services.  Such 
reviews  emphasize  identification  and  analysis  of  pat- 
terns of  patient  care  in  order  to  maintain  consistent 
.high  quality.    The  review  is  accomplished  by  consider- 
ing data  obtained  by  any  one  or  any  combination  of  the 
fol lowing: 

(i)    By  use  of  services  and  facilities  of  external  organi- 
zations which  compile  statistics,  design  profiles,  and 
produce  other  comparative  data;  or 

(iii)    By  internal  studies  of  medical  records. (10) 
It  was  expected  that  hospitals  would  respond  to  the  flexibility  allowed 
in  the  regulations  by  developing  innovative  plans  for  meeting  the  require- 
ments.   This  was,  however,  not  to  be.    The  lack  of  specificity  in  the 
regulation,  as  well  as  the  fact  that  failure  to  carry  out  reviews  was  not 
a  sufficient  cause  to  bar  a  hospital  from  Medicare  participation  (the 
"substantial  compliance"  doctrine),  meant  that  medical  audits  were  not 
likely  to  be  carried  out  except,  perhaps,  in  the  best  hospitals. 

The  1972  amendments  to  the  Social  Security  Act  (P.L.  92-603) 
contained,  among  other  provisions,  sections  establishing  the  PSRO  pro- 
gram and  revising  utilization  review  requirements  for  participation  in 
Medicare  and  Medicaid.    The  purposes  of  the  UR  revisions  were:  a)  to 
make  review  procedures  more  objective,  efficient,  and  comprehensive; 
b)  to  require  the  same  types  of  review  under  both  Medicare  and  Medicaid; 
and  c)  to  establish  review  procedures  compatible  with  the  PSRO  program. 


The  regulations  published  in  1974  pursuant  to  P.L.  92-603 

specified  the  procedures  of  medical  audit  and  formally  introduced  the 

term  "medical  care  evaluation  studies. " (11 ) 

(j)    Standard:  Medical  care  evaluation  studies. 
Medical  care  evaluation  studies  are  performed 
'to  promote  the  most  effective  and  efficient 
use  of  available  health  facilities  and  services 
consistent  with  patient  needs  and  professionally 
recognized  standards  of  health  care.  Studies 
emphasize  identification  and  analysis  of  patterns 
of  patient  care,  and  suggest  possible  changes  for 
maintaining  consistently  high  quality  patient  care 
and  effective  and  efficient  use  of  services. 
Each  medical  care  evaluation  study  (whether  medi- 
cal or  administrative  in  emphasis)  identifies  and 
analyzes  factors  related  to  the  patient  care  rendered 
in  the  facility,  and  where  indicated,  results  in 
recommendations  for  change  beneficial  to  patients, 
staff,  the  facility,  and  the  community.  Studies 
on  a  sample  or  other  basis  must  include,  but  need  not 
be  limited  to:  admissions,  durations  of  stay,  ancil- 
lary services  furnished  (including  drugs  and  biologi- 
cal s ) ,  and  professional  services  performed  on  the 
hospital  premises.    At  least  one  study  must  be  in 
progress  at  any  given  time,  and  at  least  one  study 
must  be  completed  each  year.    The  study  will  be  accom- 
plished by  considering  and  analyzing  data  obtained 
from  any  one  or  a  combination  of  the  following  sources. 

(1)  Medical  records  or  other  appropriate 
hospital  data; 

(2)  External  organizations  which  compile 
statistics,  design  profiles,  and  produce 
other  comparative  data;  and 

(3)  By  cooperative  endeavor  with  the  PSRO, 
fiscal  intermediary(ies) ,  providers  of 
services  or  other  appropriate  agencies. 

The  group  or  committee  shall  document  the  results  of 
each  medical  care  evaluation  study  and  how  such  results 
have,  where  appropriate,  been  used  to  institute  changes 
to  improve  the  quality  of  care  and  promote  more  effec- 
tive and  efficient  use  of  facilities  and  services. (12) 
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Aspects  of  these  regulations  — 


though  not  the  MCE  section  - 


were  challenged  in  court  by  the  American  Medical  Association,  so  that 
these  rules  were  not  implemented  as  planned.    They  constitute,  however, 
the  most  recent  legal  basis  for  the  MCE  program.    As  PSROs  are  granted 
conditional  authority,  they  assume  on  a  hospital  by  hospital  basis  the 
responsibility  to  carry  out  MCEs  or  to  delegate  this  function  to  the 
hospital . 

A  more  advanced  version  of  the  regulations  was  proposed  in 
1976,  but  thus  far  has  not  been  adopted.    It  is  included  here  to  show 
the  latest  thinking  of  the  Department: 

(5)    Medical  Care  Evaluation  Studies. 

(i)  The  medical  staff  shall  be  responsible  for  the 
performance  of  medical  care  evaluation  studies  to 
promote  the  most  effective  and  efficient  use  of  avail- 
able health  facilities  and  services  consistent  with 
patient  needs  and  professional  recognized  standards 

of  health  care. 

(ii)  The  studies  shall  emphasize  identification  and 
analysis  of  patterns  of  patient  care,  and  recommend 
appropriate  changes  for  maintaining  consistently  high 
patient  care  and  effective  and  efficient  use  of  ser- 
vices. 


(iii)  Each  medical  care  evaluation  study  shall  be  ■ 
designed  by  the  physician  members  of  the  committee 
and  shall  utilize  criteria  and  standards  established 

by  those  members.    The  studies  may  relate  to  admission, 
duration  of  stay,  ancillary  services  (including  the 
furnishing  of  drugs  and  biologicals)  and  professional 
services  performed  on  hospital,  premises.    The  studies 
may  be  directed  to  the  process  or  outcome  of  patient 
care  or  to  the  organizational  and  administrative 
arrangements  for  providing  such  care. 

(iv)  The  committee  shall  analyze  the  finding  of  each 
study  and  make  written  recommendations,  where 
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appropriate,  for  corrective  actions  in  the  organi- 
zation and  administration  of  health  care  delivery 
in  order  to  improve  the  quality  of  care,  promote 
more  effective  and  efficient  utilization  of  facili- 
ties and  services,  or  to  make  changes  beneficial 
to  patients,  staff,  the  facility  and  the  community. 

(v)  At  least  one  study  shall  be  in  progress  at  any 
given  time  and  at  least  one  study  must  be  completed 
each  year.    These  studies  may  utilize  data  and  infor- 
mation derived  from  the  records  of  the  institution; 
statistics,  profiles  and  comparative  data  on  hospital 
utilization  compiled  by  other  organizations  and 
reports  and  other  information  obtained  from  fiscal 
intermediaries,  providers  of  services  or  other 
agencies. 

(vi)  The  committee  shall  document  the  results  of 
each  medical  care  evaluation  study  and  how  such 
results  have  been  used  to  institute  changes  in  the 
methods  of  care  and  to  promote  more  effective  and 
efficient  use  of  facilities  and  services. (13) 

At  the  same  time  that  MCEs  became  a  legal "requirement  for  Medi- 
care and  Medicaid  participation,  the  Joint  Commission  (JCAH)  established 
them  as  a  standard  for  hospital  accreditation. (14)   As  of  July  1,  1975, 
any  hospital  surveyed  by  the  JCAH  had  to  display  two  completed  MCEs  of 
acceptable  quality  and  two  in  progress.    This  number  was  changed  in  1976 
to  be  proportionate  to  hospital  size,  ranging  from  4  to  12  per  hospital 
each  year.    The  same  numerical  requirement  was  adopted  by  the  PSRO  pro- 
gram in  January  of  1977.    PSRO  and  JCAH  requirements  for  MCEs  are  being 
made  increasingly  compatible  in  numbers,  process,  and  topics,  so  that 
MCEs  are  receiving  coordinated  support  as  tools  for  improving  quality 
of  care. 
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Conclusions.    This  section  has  attempted  to  portray  the 
historical  basis  for  the  development  of  the  MCE  component  of  the  PSRO 
program.    The  movement  to  reform  medicine  from  within  at  the  beginning 
of  the  century  gave  birth  to  the  medical  audit.    The  concern  with  cost 
and  quality  in  the  1960s  and  1970s  gave  it  new  importance.    The  contri- 
bution of  MCEs  to  improvement  of  hospital  care  is  the  subject  of  this 
volume. 
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2.2  Purpose  of  Medical  Care  Evaluation  Studies 

The  Medical  Care  Evaluation  (MCE)  study  begins  with  a  medical 
audit  whose  purpose  is  twofold.    First,  the  audit  assesses  the  quality 
and  efficiency  of  services  rendered  by  physicians,  other  health  care 
practitioners  and  ancillary  staff  in  hospitals  and  other  institutions. 
Secondly,  it  documents  patterns  of  patient  outcomes;  i.e.,  the  status 
of  patient  health  after  the  intervention  of  medical  care.    The  MCE  method- 
ology established  to  accomplish  these  purposes  is  actually  a  flexible 
set  of  requirements.    These  requirements  will  be  discussed  in  detail  at 
a  later  point.    It  should  be  noted  now,  however,  that  this  methodology 
relies  on  the  development  of  criteria.    These  criteria  objectively  and 
rigorously  describe  local  expectations  of  how  patients  should  be  admitted, 
diagnosed,  treated,  and  released  from  institutions.    PSROs  then  test  these 
cirteria  against  medical  records  (such  as  patient  charts).    The  notation  of 
the  presence  or  absence  of  these  criteria  in  these  charts  serves  as  the  data 
upon  which  judgments  are  made  as  to  the  quality  and  efficiency  of  the  process 
and  outcomes  of  medical  care.    If  variations  are  noted  with  respect  to  the 
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criteria,  the  PSRO  may  identify  deficiencies  in  the  medical  care  pro- 
cess and/or  patient  outcomes.    Local  medical  staffs  are  then  responsible 
for  eliminating  these  deficiencies  by  suggesting  and  implementing  correc- 
tive actions  such  as  modifying  practioner  behavior  through  mass  continu- 
ing medical  educational  programs,  counseling  individual  practitioners, 
or  altering  facility  policies.    Although  these  steps  are  executed  in 
most  types  of  medical  audit,  the  MCE  methodology  is  distinguished  by 
the  feedback  loop  which  is  designed  to  insure  that  the  actions  recommended 
in  the  initial  audit  are  actually  carried  out.    Through  a  series  of  follow- 

a 

up  steps  including  reaudit,  MCEs  assess  the  extent  to  which  there  is  actual 
improvement  in  medical  care  or  in  patient  outcomes  after  the  corrective 
actions  have  been  taken. 

The  methodology  of  the  MCE  is  similar  to  the  explicit  process 
approach  described  by  Brook  in  1973.    The  explicit  process1  approach 
primarily  analyzes  the  process  of  medical  care;  i.e.,  the  way  in  which 
patients  are  handled  in  institutions.    Yet,  MCEs  attempt  to  go  deeper 
in  their  analysis  of  medical  care  by  assessing  patient  health  status  at 
the  time  of  discharge.    To  do  this,  outcome  criteria  are  developed  which 
establish  standards  for  discharge  against  which  actual  patient  health 
status  is  measured.    The  use  by  PSROs  of  both  the  explicit  process  and 
outcome  approach  will  undoubtedly  continue  as  physician  involvement, 
particularly  in  setting  criteria,  is  requisite  in  these  approaches.  The 
involvement  by  those  actually  responsible  for  providing  medical  care 
is  a  primary  way  in  which  appropriate  action  to  improve  medical  care 
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process  and  patient  outcomes  can  be  assured. 

Because  of  difi".  ulties  in  using  techniques  such  as  explicit 
3 

group  outcome  estimations,    health  status  indexes  have  not  been  incor- 
porated into  the  MCE  procedure.    Basically,  "a  major  problem  in  the 
development  of  a  health  index  is  a  lack  of  a  conceptual  definition  of 
health  capable  of  being  translated  into  operational  terms."4  Thus, 
not  all  aspects  of  medical  care  are  presently  being  studied  by  MCEs. 
The  attempt  has  been  made  to  broadly  define  medical  care  as  the  appli- 
cation of  physician  and  other  health  care  practitioner  services,  as 
well  as  ancillary  services  to  patients  for  the  purpose  of  improving 
health  status.    The  favorable  application  of  medical  care  results  in 

positive  patient  outcome. 

5 

Donabedian    states  that  good  medical  care  has  the  following 
foud  major  characteristics: 

Efficiency 
Qual ity 
Continuity 
Access ibi 1 ity 

He  breaks  down  the  characteristic  of  "Quality"  into  the  areas  of  pro- 
fessional competence,  qualitative  adequacy  in  medical  knowledge  or 
surgical  skill,  and  patient  acceptance  of  the  care  rendered.    Of  these 
three  areas,  MCEs  have  primarily  concentrated  on  assessing  the  knowledge, 
skills,  and  competence  of  practitioners  in  the  hospital.    Also,  they 
have  attempted  to  evaluate  the  necessity  and  efficiency  of  admissions 
to  those  hospitals.    Thus,  in  addition  to  their  assessment  of  patient 
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outcomes  discussed  earlier,  MCEs  have  focused  primarily  on  the  effi- 
ciency and  quality  (minus  the  area  of  patient  acceptance  of  care  rendered) 
aspects  of  good  medical  care  in  hospitals.    It  is  apparent,  then,  that 
the  assessment  of  the  continuity  and  accessibility  of  medical  care  are 
not  yet  a  part  of  PSRO  MCE  activity.    This  is  true  primarily  because  of 
the  fact  that  MCE  activity  has  followed  the  implementation  of  PSRO'  review, 
which  has  been  mostly  of  acute  care  facilities. 

Continuity  and  accessibility  of  care  are  concepts  which  go 
beyond  the  inpatient  taking  on  a  greater  importance  for  the  ambulant 
and/or  chronically  ill.    While  these  two  concepts  have  not  been  given 
any  significant  amount  of  attention  by  MCEs  to  date,  demonstration 
projects  in  ambulatory  and  long-term  care  facilities  are  now  concentra- 
ting more  on  these  characteristics  of  good  health  care.    For  the  purpose 
of  this  report,  however,  MCEs  will  be  evaluated  for  their  ability  to  improve 
assessable  medical  care  as  defined  by  Roemer.6    His  characterization  of 
medical  care  is  that  hospital  care  which  is  actually  delivered  as 
opposed  to  all  forms  of  health  care  theoretically  available.    It  should 
be  no  surprise,  then,  that  MCEs  now  serve  primarily  to  identify  problem 
areas  in  medicine,  surgery,  and  hospital  administration  as  they  exist  1n 
hospitals.    Future  evaluations  of  MCEs  will  include  the  analysis  of  MCE 
activity  in  alj_  areas  of  health  care. 

Results  of  MCEs 

Some  final  definitions  are  in  order.    The  initial  audit  of 
the  MCE  identifies  patterns  of  medical  care  and  patient  outcomes.  When 
these  patterns  are  compared  to  written  criteria,  deficiencies  in  medical 
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care  or  patient  outcomes  may  be  noted.    One  or  several1 deficiencies  may  exist 
in  problem  areas7  which  could  occur  across  several  diagnostic  categories. 
These  problem  areas  increase  in  importance  as  they  affect  a  greater  number  of 
patients  and  practitioners,  and  cause  higher  capital  outlays.    The  results  of 
the  initial  audit  are  reflected  by  the  following  categories: 

Number  of  deficiencies  found  in  medical  care 
Number  of  deficiencies  found  in  patient  outcomes 
Number  of  problem  areas  revealed  in  medical  care 

and  patient  outcomes 
Description  of  deficiencies 
Description  of  problem  areas 
Significance  of  deficiencies 
Importance  of  the  problem  areas 
Type  of  recommended  actions  suggested 
Number  of  recommended  actions  made 

The  proximate  results  of  the  MCE  are  determined  after  the  reaudit 

is  complete  and  its  results  are  compared  to  the  results  of  the  initial 

audit.    The  proximate  results  appear  in  the  following  categories: 

Number  of  deficiencies  eliminated  in  medical  care 
Number  of  deficiencies  removed  in  patient  outcomes 
Number  of  problem  areas  eliminated  in  medical  care 
Number  of  problem  areas  eliminated  in  patient  outcomes 
Significance  of  removing  deficiencies 
Significance  of  removing  problem  areas 
Number  of  recommendations  implemented  by  the  time  of 
reaudit 

The  existence  of  proximate  results  indicates  whether  or  not  the  audit/ 
follow-up/reaudit  cycle  has  been  immediately  successful  in  improving 
medical  care  or  patient^ outcomes.    If  it  has  improved  medical  care  or 
patient  outcomes,  then  the  MCE  had  impact.    Only  when  the  benefits  of  the 
changes  are  analyzed  according  to  more  long-term  parameters  such  as 
improvement  in  patient  health  status,  improvement  in  the  health  status 
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of  the  population,  or  improvement  in  the  costs  or  efficiency  of  medi- 
cal care,  however,  can  the  final  results  of  the  MCE  be  truly  described 
and  assessed  for  their  value. 

The  Role  of  BQA  in  Medical  Care  Evaluation  Studies 

BQA  has  been  responsible  for  organizing  PSROs  from  groups  of 
physicians  across  the  entire  country.    It  has  been  the  responsibility 
of  the  Bureau  to  encourage  these  PSROs  to  begin  retrospective  reveiw  as 
soon  as  it  became  possible.    To  do  this,  the  Bureau  has  attempted  to 
develop  and  encourage  MCE  compliance.    It  has  also  offered  technical 
assistance  in  relation  to  the  completion  of  MCEs  to  PSRO  staff  in  order 
to  facilitate  the  implementation  of  MCE  review.    Finally,  it  has  been 
the  responsibility  of  the  Bureau  to  evaluate  the  success  with  which  the 
program  is  implemented  and  the  utility  of  the  entire  MCE  program. 

To  establish  the  MCE  policy,  BQA  issued  the  first  draft  of 
the  MCE  Transmittal  (PSRO  Transmittal  No.  43)  in  July  of  1976.  The 
transmittal  defined  the  MCE  and  set  forth  the  requirements  to  be  met 
in  its  completion.    In  addition,  it  stated  the  overall  requirements 
a  PSRO  was  asked  to  meet  in  establishing  a  regional  MCE  program.  The 
relative  roles  of  the  hospitals  and  the  PSRO  in  meeting  these  require- 
ments were  included  in  the  Transmittal  and  it  was  approved  by  the 
National  Professional  Review  Council  in  January  of  1977  with  issuance 
being  final  on  January  25,  1977. 

The  following  is  the  definition  of  a  medical  care  evaluation 
(MCE)  study  as  set  forth  in  the  Transmittal: 
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Definition 


Medical  Care  evaluation  (MCE)  studies  are  a  form  of 
health  care  review  in  which  an  in-depth  assessment  is 
made  of  the  quality  of  the  delivery  and  organization 
of  health  care  services.    They  are  designed  to  assure 
that  1)  health  care  services  are  appropriate  to  the 
patient's  needs  and  are  of  optimal  quality  and  2)  health 
care  organization  and  administration  support  the  timely 
provision  of  quality  care.    By  systematically  and  objec- 
tively documenting  patterns  of  health  care,  studies 
should  be  capable  of  identifying  deficiencies  in  the 
quality  of  that  care  so  that  where  they  exist,  specific 
improvement  programs  may  be  directed  at  the  causes  of  the 
deficiencies.    Study  data  may  be  collected  either  retro- 
spectively or  concurrently,  but  data  analysis  is  always 
performed  retrospectively. 

MCE  studies  are  not  clinical  trials  or  clinical  research. 
Their  purpose  is  to  evaluate  current  local  health  care 
practices  to  determine  if  they  meet  current  optimal 
achievable  professional  standards  of  acceptability.  A 
successful  MCE  study  does  not  set  out  to  expand  medical 
knowledge,  although  expansion  of  knowledge  may  well  be 
a  by-product  of  the  study.    Rather,  the  primary  goal  of 
the  MCE  process  is  to  cause  improvement  in  care  by  assur- 
ing that  currently  generally  accepted  or  proven  measures 
are  being  optimally  utilized  by  health  care  practitioners 
and  providers,  given  the  local  resources  available. 

The  Transmittal,  in  defining  the  Bureau  policy  regarding  what 
constitutes  an  acceptable  MCE,  then  delineated  the  10  essential  charac- 
teristics of  an  acceptable  MCE  study.    These  characteristics  were  pur- 
posefully generic  to  ensure  that  institutions  could  perform  their  review 
with  a  minimum  amount  of  constraint  while  insuring  that  MCE  studies  done 
throughout  all  PSRO  areas  would  have  common  features  and  structural  inte- 
grity.   The  10  characteristics  which  must  be  met  in  the  performance  of 
an  MCE  are  as  follows: 

1.      it  focuses  upon  a  known  or  suspected  problem  area 
impacting  on  the  quality  of  health  care; 


-29- 


it  focuses  on  a  well  defined  topic  and  is  carried 
out  in  accordance  with  objectives  explicity  stated 
in,  and  specifically  developed  for,  the  study; 

it  utilizes  written  criteria  against  which  actual 
patterns  of  health  care  practice  are  compared; 

it  provides  for  collection  of  data  on  a  sample  of 
patients,  the  size  and  composition  of  which  is 
appropriate  to  the  study  topic  and  objectives. 
The  MCE  study  identifies  patterns  of  care  related 
to  the  subject  under  study  and  is  not,  in  general, 
concerned  with  individual  patients  or  payment. 
Therefore,  the  sample  should  be  drawn  from  all 
patients  in  the  institution,  with  authorization 
from  the  institution,  and  not  be  limited  to  Title  V, 
XVIII  and  XIX  patients. 

c 

it  provides  for  thorough  peer  analysis  of  the  rea- 
sons for  any  discrepancies  between  the  written 
criteria  which  reflect  optimal  achievable  health 
care  practices,  and  the  data  collected  on  actual 
health  care  practices  to  determine  whether  varia- 
tions are  objectively  justifiable  or  represent 
problems  that  require  corrective  action; 

it  results  in  specific  written  recommendations, 
where  indicated,  to  individuals,  boards  or  com- 
mittees who  are  responsible  for  assuring  the  quality 
of  care  in  the  institution,  and  to  those  who  are 
responsible  for  effecting  changes  in  health  care 
practices,  either  through  appropriate  continuing 
health  education  activities,  through  changes  in  the 
organization  and  administration  of  health  care 
delivery,  or  through  other  means  appropriate  to 
the  deficiencies  identified  by  the  study,  in  order 
to  improve  the  quality  of  care  and  promote  more 
effective  and  efficient  utilization  of  facilities 
and  services; 

it  includes  documentation  regarding  when,  where 
and  by  whom  the  recommended  actions  were,  in  fact 
implemented;  '  ' 

it  includes  a  plan  for  follow-up  evaluation,  where 
indicated,  to  determine  what  changes  have  occurred 
as  a  result  of  actions  recommended  pursuant  to  sub- 
paragraph (6)  to  correct  specific  deficiencies 
identified  by  the  study; 
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9.     it  provides  for  such  a  follow-up  evaluation, 
where  indicated,  to  be  completed  in  a  reason- 
able time,  but  usually  no  later  than  one  year 
following  the  performance  of  the  initial  study. 
The  follow-up  evaluation  should  be  limited  to 
relevant  key  indicators  which  will  reflect  the 
efficacy  of  the  actions  recommended;  and 

10.     it  at  least  provides  for  periodic  reporting  of 
a  summary  of  the  quality  assurance  activities 
to  the  governing  board  of  the  hospital  to  assist 
the  board  in  meeting  its  public  responsibility 
to  assure  the  provision  of  quality  care  in  its 
institution.    If  results  of  a  study  require 
specific  board  action  such  results  of  the  parti- 
cular study  must  be  reported  to  the  board. 

Together,  this  set  of  ten  characteristics  establishes  a 
feedback  loop  which  begins  with  a  medical  audit  to  identify  problems 
and  includes  a  reaudit  to  eliminate  the  deficiencies  found  in  the  initial 
audit.    The  first  six  characteristics  describe  the  actual  performance  of 
a  medical  audit.    The  first  characteristic,  the  selection  of  an  appropriate 
topic  for  study,  is  a  particularly  crucial  one  for  the  success  of  the  MCE. 
Only  if  the  topic  is  one  in  which  there  exists  problem  areas  important 
to  large  groups  of  patients  and  practitioners,  will  the  study  ultimately 
contain  significant  results.    The  other  five  initial  characteristics 
indicate  the  necessity  for  setting  criteria,  selecting  statistically  appro- 
priate samples  of  patient  charts,  abstracting  criteria-related  data 
from  these  charts,' identifying  deficiencies  through  analysis  of  the 
data,  and  recommending  actions  to  correct  these  deficiencies.    Most  of 
the  widely  used  audit  techniques  (JCAH,  AHA)  address  these  six  charac- 
teristics. 
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The  remaining  four  characteristics  set  forth  the  unique 
follow-up  requirements  of  the  PSRO  MCE.    First,  it  must  be  verified 
that  actions  recommended  after  the  initial  audit  were  actually 
implemented.    Second,  through  the  mechanism  of  reaudit,  which  is 
specified  in  characteristics  8  and  9,  actions  must  also  be  evaluated 
for  their  effectiveness  in  removing  deficiencies.    Reaudits  which  are 
to  be  performed  within  12  months  of  the  completion  of  the  initial  audit, 
again  follow  the  first  five  steps  using  either  all  criteria  or  only  those 
with  which  the  original  audit  identified  deficiencies.    The  follow-up 
steps  and  the  topic  selection  step  are  the  portions  of  the  methodology 
most  crucial  to  ensure  that  the  MCE  will  produce  useful  results.  Without 
them,  there  is  little  chance  of  converting  the  initial  findings  into  an 
actual  improvement  in  the  process  of  medical  care. 

In  addition  to  these  ten  characteristics,  the  Transmittal  set 
policy  stating  the  requirements  for  the  overall  operation  of  a  PSRO  MCE 
Program.    In  particular,  it  specified  the  number  of  MCEs  to  be  done  in 
each  PSRO  hospital.    This  was  to  be  based  upon  the  number  of  admissions 
in  the  individual  hospital,  and  consequently  ranged  between  4  and  12 
audits  per  year.    Also,  the  Transmittal  required  that  each  MCE  identify- 
ing deficiencies  of  importance  was  be  followed  by  a  reaudit.    Finally,  it 
set  forth  the  overall  goals  of  the  MCE  Program  to  be  achieved  by  each 
PSRO.    These  were:  first,  that  the  .PSRO  must  address  aJJ_  areas  of  health 
care,  not  only  those  which  are  physician  related;  second,  that  the 
patients  reviewed  should  not  only  be  ones  covered  by  Federal  funds; 
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and  lastly,  that  overall  MCE  activity  should  aim  towards  efficiently 
obtaining  significant  results. 

In  addition  to  policy  determination,  the  Bureau  is  responsible 
for  the  evaluation  and  enforcement  of  MCE  program  implementation  and 
for  ensuring  that  it  has  an  effect  on  the  health  care  system  in  the  ' 
United  States.    This  report  is  the  first  in-depth  assessment  of  the 
results  that  have  been  obtained  thus  far  from  MCE  activity.    In  addition, 
this  evaluation  attempts  to  project  the  kinds  of  results  that  are  expected 
from  the  program  in  the  near  future.    Until  now,  evaluation  has  been 
limited  to  summaries  of  the  state  of  MCE  activity  implementation.  The 
National  Professional  Review  Council  was  presented  with  the  findings  of 
an  early  evaluation  last  November  which  indicated  that  MCE  activity  was 
the  last  major  project  of  each  of  the  conditional  PSROs  following  dele- 
gation and  concurrent  review  implementation.    In  1975,  less  than  half 
of  the  conditional  PSROs  were  shown  to  be  actively  performing  and  report- 
ing their  MCE  activity  via  the  PSRO  Management  Information  System  (PMIS). 
This  system,  consisting  of  abstracts  of  all  audits  and  reaudits  sub- 
mitted for  central  Bureau  processing,  has  been  the  main  source  for 
identifying  MCE  related  information  (see  sample  abstracts  at  the 
end  of  this  section).    By  the  second  quarter  of  1976,  52  of  65  PSROs 
were  actively  performing  and  reporting  MCE  information  on  the  PMIS  system. 
The  initial  evaluation  in  1975  has  been  continued  each  quarter,  and 
is  now  being  used  to  promote  program  compliance  by  graphing  the  activity 
of  each  PSRO  in  relation  to  others.    Thus,  PSROs  are  able  to  guage  their 
own  level  of  MCE  activity  relative  to  similar  PSROs. 
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The  present  evaluation  has  been  designed  and  carried  out 
by  the  BQA  in  its  continuing  responsibility  for  evaluating  the  MCE 
program.    In  it,  nationwide  activity  is  summarized  and  categorized  by 
areas  of  medical  care.    Also,  the  compliance  of  PSROs  with  the  policy 
requirements  of  the  transmittal  which  defines  acceptable  MCE  methodo- 
logy is  explored.    In  particular,  the  crucial  steps  of  topic  selection 
and  follow-up  are  assessed  as  indicators  of  the  success  of  the  MCE  pro- 
gram. Finally,  the  evaluation  focuses  on  the  impact  which  MCEs  have 
had  to  date  in  four  major  areas  of  medicine  and  surgery.    As  recommended 
by  the  Institute  of  Medicine  Study,  this  analysis  of  impact  is  based 
on  "short  term  outcome  measures  which  can  be  analyzed  closer  in  time  to 
the  provision  of  care  than  final  end  result  measures."    The  plans  for 
future  long-term  evaluation  of  MCEs  have  been  developed  in  part  and  will 
be  discussed  at  the  end  of  this  volume.    It  is  likely  that  health  status 
indicators  will  be  used  in  evaluations  scheduled  to  take  place  over  the 
next  several  years. 

Role  of  PSROs  in  Medical  Care  Evaluation  Studies 

PSROs  have  at  least  five  areas  of -responsibil ity  respective 

to  MCE  activity.    The  Transmittal  specifies  that  they  are  to  ensure 

the  following: 

1.     The  actual  performance  of  MCEs.    It  is  their 
responsibility  to  perform  audits  and  follow- 
up,  including  reaudits  in  each  of  its  hospitals 
which  is  nondelegated  (i.e.,  not  charged  with 
the  responsibility  of  reviewing  its  own  Federal 
patients).    The  PSRO  is  responsible  for  perform- 
ing the  same  number  of  quality  MCEs  in  the 
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nondelegated  hospital  as  defined  in  the  Trans- 
mittal.   In  addition,  the  PSRO  is  responsible 
for  organizaing  areawide  MCEs,  an  activity  in 
which  several  hospitals  perform  a  single  MCE 
of  the  same  topic  with  the  same  criteria  on  their 
own  patients. 

2.  Monitoring  MCE  activity  in  delegated  hospitals. 
The  PSRO  is  responsible  for  gauging  the  appropriate 
amount  of  MCE  activity  in  each  of  the  delegated 
hospitals.    It  is  also  responsible  for  reviewing 
these  studies  for  structural  integrity  and  to 
ensure  that  the  studies  are  actually  having  impact 
on  patient  care. 

3.  Delegating  Review  Activity.    The  PSRO  may  delegate 
Us  review  responsibility  to  hospitals  which  prove 
themselves  interested  and  proficient  in  the  perform- 
ance of  medical  audit.    Once  the  determination  is 
made,  the  PSRO  is  responsible  for  monitoring  this 
activity  for  adequate  quantity  and  quality  of  studies. 
Delegation  of  MCE  activity  is  independent  of  the 
delegation  of  Concurrent  Review. 

4.  Reporting  all  activity  to  BQA.    The  PSRO  is  responsible 
for  reporting  all  activity  by  means  of  the  PSRO  Manage- 
ment Information  System.    For  each  MCE  study  completed 
in  a  delegated  or  nondelegated  hospital,  an  MCE  Study 
Abstract  is  completed.    A  similar  reaudit  is  submitted 
to  BQA  at  the  completion  of  each  reaudit.    Each  MCE 
Study  Abstract  (see  attached  Form  131)  contains  infor- 
mation on  the  exact  MCE  methodology  used,  the  amount 

of  time  and  manpower  required  to  perform  the  studies, 
the  problem  areas  of  medicine,  surgery,  or  hospital 
administration  under  review,  the  type  of  patients  being 
audited,  and  the  integrity  of  the  MCE  study  structure. 
The  forms  also  contain  information  on  the  results  obtained 
from  the  initial  audit.    These  results  take  the  form  of 
the  number  of  deficiencies  identified,  and  the  number 
and  type  of  recommended  actions  taken  to  correct  these 
deficiencies.    The  Restudy  (Reaudit)  form  PMIS  133 
gives  similar  information  on  the  steps  taken  to  perform 
the  reaudit.    Further  results  are  identified  on  these 
by  the  notation  of  whether  or  not  deficiencies  have 
been  removed  by  the  time  of  the  reaudit.    All  of  the 
results  on  the  PMIS  forms  are  called  proximate  results 
These  results  are  indicative  of  conclusions  which  can 
be  drawn  about  medical  care  and/or  patient  outcomes. 
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But  the  importance  of  these  conclusions  and 
their  effect  on  patients  cannot  be  accurately 
determined  from  proximate  results  alone. 

A  factor  limiting  evaluation,  at  this  point  in  time,  relates 
to  reporting  requirements.    It  is  the  responsibility  of  the  PSRO  to 
submit  each  of  its  131  and  133  forms  45  days  after  the  end  of  the  quar 
ter  in  which  the  audit  or  reaudit  was  completed.    The  forms  are 
processed  by  the  PMIS  contractor  and  checked  for  completeness  and 
consistency.  Many  of  the  forms  have  been  returned  to  the  PSROs  for  cor 
rection  and  completion  which  causes  delay  in  their  entrance  into  the 
overall  PMIS  data  base.    It  must  be  emphasized  that  the  time  from  com- 
pletion of  an  audit  or  reaudit  to  the  time  of  its  entry  into  the  PMIS 
data  base  can  be  as  much  as  7  months.    Because  information  from  these 
forms  serve  as  the  initial  source  of  data  for  this  evaluation,  this 
evaluation  limits  much  of  its  analysis  to  MCEs  completed  by  December 
of  1976. 
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MCE  PROGRAM  ACTIVITY  SUMMARY 

Activity  Report  on  the  Medical  Care  Evaluation  Component 
of  the  PSRO  Program 


Objective 

This  section  reports  and  describes  information  on  the  quantity, 
distribution,  and  subject  content  of  MCE  audits.    For  the  purpose  of  this 
report,  an  audit  is  defined  as  that  part  of  an  MCE  which  is  completed 
through  the  sixth  step;  "specific  recommendations"  (see  diagram  I).  Dis- 
cussion of  the  concluding  MCE  steps  7-10,  known  as  the  reaudit  stages, 
is  found  in  Chapter  V. 
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Data  Sources  and  Validity 

The  PSRO  Management  Information  System  (PMIS)  has  been  gather- 
ing data  on  MCE  activity  since  the  beginning  of  the  program  in  1975. 
Reporting  by  the  PSROs  is  quarterly  through  the  use  of  the  following  three 


forms : 


•  BQA  131  -  The  MCE  Study  Abstract,  transmitted  by 

the  PSRO  for  each  MCE  completed  through 
the  analysis  stage. 

•  BQA  133  -  The  MCE  Restudy  Abstract,  transmitted 

after  a  restudy  is  performed. 

•  BQA  135  -  The  MCE  Study  Status  Report,  reports  the 

status  of  each  MCE  underway  in  a  given 
quarter,  regardless  of  its  completion 
stage. 

There  is  no  attempt  by  BQA  to  validate  the  data  elements  accumulated 
through  PMIS.    Forms  received  by  the  PMIS  contractor,  however,  are 
checked  for  completeness  and  consistency;  problems  are  returned  to 
the  PSRO  for  resolution.    The  quality  of  the  data  for  this  study  is 
further  influenced  by  several  time  dependent  factors: 


1 


Under  the  PSRO  Program,  hospitals  are  required  to 
complete  specified  numbers  of  MCEs.  While  it  is 
to  their  interest  to  promptly  submit  quarterly  " 
reports,  (the  final  due  date  for  these  reports  is 
45  days  post  quarter  end),  there  is  considerable 
lag  in  meeting  reporting  requirements. 

Data  processing  procedures  can  lead  to  lags  of 
several  months  before  received  PMIS  forms  are 
analyzed.    Such  lags  are  randomly  distributed 
and  thus  do  not  contribute  any  known  bias. 

When  incomplete  forms  are  returned  to  PSROs  for 
additional  information,  there  usually  is  an  additional 
delay  of  1  1/2  months. 
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Health  Services  Administration 


MEDICAL  CARE 
^EVALUATION  STUDY  ABSTRACT 


Office  of  Management  and  Budget 
Approval  Number  68-R 1476 
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HOSPITAL 
ID  NO. 


0  71 


PSRO  NAME 

ROCKVTLLE  PSRO 

PSRO  NO. 

842 

MCE  STUDY  BEGUN 
Month      07  Year74 

MCE  STUDY  COMPLETED  THROUGH  STEPS 
BELOW       Month  09            Year  ?4 

CONDUCTED  BY:     □  PSRO 

K]  DELEGATED  HOSPITAL 


STUDY  TOPIC 


Management  of  patients  with 


(Write  in)  suspected  AMI  within  first  hour 

after  admission  to  emergency  room 

2.      METHOD  FOR  SELECTING  STUDY  (Check  One) 

□  a.  PROFILE  ANALYSIS 

□  b.  CONCURRENT  REVIEW 

□  c.  OTHER  MCE  STUDY 

□  d.  ANALYSIS  OF  MEDICAL  RECORDS 
gC]  e.  PERCEIVED  NEED 

□  f.  OTHER  (Specify)  


3.      TOPIC  CHARACTERISTIC  (Check  One) 
□  a.  INCIDENCE/PREVALENCE 

PREVENTABLE  MORBIDITY/MORTALITY 
COST  OF  CARE 

OTHER  (Specify)  


E3  b- 

□  c 

d: 


4.  HOSPITAL  STUDY  RESPONSIBILITY  (Check  One) 

gj  a.  MEDICAL  AUDIT  COMMITTEE 

□  b.  UR  COMMITTEE 

□  c.  SERVICE/DEPT.  (Specify)  


□  d.  OTHER  (Specify)  

5.      NON-PHYSICIAN  PARTICIPATION  (Specify) 


E.R.  Nursing 


Respiratory  therapy 


X-ray  technologists 


6.      DERIVATION  OF  CRITERIA  (Check  One) 
0  a.  SELF  GENERATED 

□  b.  NATIONAL  ORGANIZATION 

□  c.  OTHER  PSRO(S) 

□  d.  OTHER  HOSPITAL(S) 

□  e.  OTHER  (Specify)  


5-76 


?A- 1 3  I 


MCE  STUDY  ID  NO. 
16 


7.      STUDY  SITE  (Check  One) 

□  a.  PSRO-WIDE 

□  b.  PSRO  SUB-AREA  (NO.  HOSPITALS, 
c.  INDIVIDUAL  HOSPITAL 

□  d.  SERVICE  /DEPT.  OF  HOSP.  (Specify). 

□  e.  OTHER  (Specify)  


8.  TYPE  OF  DATA  COLLECTION  (Check  One) 

□  a.  RETROSPECTIVE 

□  b.  CONCURRENT 
0  c.  MIXED 

9.  SAMPLE  CHARACTERISTICS 

50 


a.  NUMBER  OF  SUBJECTS  

b.  POPULATION  100%  sample 


10. 


c  other   Sil  Pts-  adm.  to  ER  with  suspected  MI 
DATA  INSTRUMENT  (Check  One)    10  ^  3 
Kj  a.  SPECIAL  FORM 

□  b.  ROUTINE  FORM 

□  c.  OTHER  (Specify)  

11.  DATA  QUALITY  CONTROLS  (Check  all  that  apply) 
0  a.  VERIFIED 

□  b.  VERIFIED  -SOURCE  INADEQUATE 

□  c.  VERIFIED -DATA  INACCURATE 

□  d.  NONE 

□  e.  OTHER  (Specify)  

12.  DATA  PROCESSING  (Check  One) 
@  a.  MANUAL 

□  b.  EDP 

□  c.  MIXED 

EDP  Cost:  S  

13.  FINDINGS  (Check  One) 

□  a.  COMPLIANCE  WITH  STANDARDS  (Go  to  item  20) 
b.  VARIATION  FROM  STANDARDS 

□  (1)     VARIATION  JUSTIFIED  (Go  to  item  20) 

□  (2)  DEFICIENCY  IDENTIFIED  (Proceed  with  item  14) 
0  (3)     MORE  THAN  ONE  DEFICIENCY  (Proceed  with 

item  14,  specify  ruimbar  I  ) 


Continued  on  reverse  side 


14.        DEFICIENCY  ANALYSIS  (Check  One) 

□  a.  KNOWLEDGE 
H  b.  PERFORMANCE 

□  c.  OTHER  


16. 


ATTRIBUTION  (Check  all  that  apply) 

□  a.  ONE  HOSPITAL 

□  b.  MORE  THAN  ONE  HOSPITAL  (NO.. 

□  c.  INDIVIDUAL  PHYSICIAN 
[3f  d.  GROUP(S)  OF  PHYSICIANS 
[%  e.  NON-PHYSICIAN(S) 

f_2F  f-  ADMINISTRATIVE 

□  g.  FURTHER  STUDY  NEEDED 

□  h.  OTHER  


TYPE  OF  ACTION  RECOMMENDED  (Check  all  that 
apply  and  describe) 

□  a.  PRACTITIONER  COUNSELLING  


b.  EDUCATIONAL  PROGRAM 


efficacy  of  im- 
sinoi  E  HnsPiTAi    medicce  monitoring 
of  cardiac  rhythm 


□  (2)     PSRO  SUB-AREA 

□  (3)     PSRO -WIDE  

§  c  administrator  HHAMfip  organize  an 


emergency  team  for  use  in  ER 

□  d.  CHANGE  IN  CONCURRENT  REVIEW 


□  e.  OTHER  REVIEW  MODIFICATION 


□  f.  OTHER 


17. 
18. 


ESTIMATED  RE-STUDY  DATE  (mo/yr)        Jan  1975 


IF  EDUCATION  IS  RECOMMENDED,  IS  THERE 
LINKAGE  WITH  EXISTING  CONTINUING  MEDICAL 
EDUCATION  ACTIVITIES? 
(3  yes  □  no 


IF  YES,  PLEASE  DESCRIBE: . 


19. 


RESPONSIBILITY  FOR  ACTION  (Check  all  that  apply) 
C?  a.      MEDICAL  STAFF  (COMMITTEE) 

□  b.      HOSPITAL  BOARD  OF  TRUSTEES 
2j  c.      HOSPITAL  ADMINISTRATION 

□  d.      SERVICE/DEPT.  (Specify)  


□  e.      OTHER  (Specify)  

20.        PERSON  HOURS  UTILIZED  (Round  to  half  hours) 


MCE  STUDY  TASK 

PHYSICIAN 

OTHER 

a. 

0 

SELECTION  AND  DESIGN 

1 

1 

b. 

SETTING  CRITERIA 
AND  STANDARDS 

3 

c. 

DATA  COLLECTION 
AND  DISPLAY 

28 

d. 

INTERPRETATION  AND 
ANALYSIS  OF  FINDINGS 

2 

2 

e. 

TOTAL 

6 

31 

ADDITIONAL  EXPLANATORY  REMARKS  (USE  ADDITIONAL  PAPER  IF  NECESSARY) 

13b.  Deficiencies  identified: 

1)  Lack  of  E.R.  staff  trained  in  handling  of  suspected  A.  M.  I.  patients  resulting  in 
unnecessary  delays  in  initiating  monitoring  of  cardiac  rhythm. 

2)  Delays  in  obtaining  patient  chest  X-rays  fnam  radiology  department. 

16  c.  (continued)  :  return  chest  X-rays  to  emergency  room  for  review  by  physician  team 
immediately  to  assist  in  determining  further  course  of  treatment. 
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Bureau  of  Quality  Assurance 
I  lealth  Services  Administration 


RE-STUDY  REPORT 


Office  of  Management  and  Budget 
Approval  Nur^r^8-R1476 


HOSPITAL 
ID  NO. 


071 


FSRO  NAME 

Rockville  PSRO 

PSRO  NO. 
842 

DATE  RESTUDY  BEGAN 
Month    ol            Year  75 

DATE  RESTUDY  COMPLETED 
Month  02  Year75 

CONDUCTED  BY: 


□  PSRO 

^delegated  HOSPITAL 


1.  IDENTIFICATION  OF  STUDY  _ 
a.  TOPIC  OF  MCE  STiinY      Patients  with  AMI  in 


first  hour  after  admission  to  ER 


b.  ABSTRACT  ID  NUMBER  

c.  DATE  STUDY  COMPLETED  THROUGH  RECOMMENDED 
ACTIONS  09/74  

2.  RESTUDY  METHODOLOGY 

a.  TYPE  OF  DATA  COLLECTION  (Check  One) 
0(1)  RETROSPECTIVE 

□  (2)  CONCURRENT 

□  (3)  MIXED 

b.  SAMPLE  CHARACTERISTICS 

(1)  NUMBER  OF  SUBJECTS  20  

(2)  pciphi  ation    100%  sample  

(3)  OTHER  DESCRIPTION  


0 


c.  DATA  INSTRUMENT  (Check  all  that  apply) 

□  (1)    SPECIAL  FORM 

□  (2)     ROUTINE  FORM  -  SPECIAL  ELEMENTS 

□  (3) 
(4) 


ROUTINE  FORM  -  ROUTINE  ELEMENTS 

none 


ELEMENT(S)  USED. 


(5)     ELEMENT(S)  TO  BE  INCORPORATED  IN 
ROUTINE  DATA  SET  


d.  CRITERIA  AND  STANDARDS  (Check  One) 
S(D     SUBSET  OF  ORIGINAL 

□  (2)     ORIGINAL  SET 

□  (3)     OTHER  (Specify)  

e.  MODIFICATIONS  OF  CRITERIA  AND  STANDARDS 

(Check  One) 

□  (1)  SUBSTANTIAL 

□  (2)  SOME 
DJ(3)  NONE 


3.  FINDINGS 

a.  ORIGINAL  FINDINGS  (Chsck  One) 

IE]  (1)     ACTIONS  CORRECTED  DEFICIENCIES 

□  (2)     ACTIONS  PARTIALLY  CORRECTED 

DEFICIENCIES 

□  (3)    ACTIONS  PRODUCED  NO  CHANGE 

□  (4)     ACTIONS  NOT  IMPLEMENTED 

b.  NEW  PROBLEM  IDENTIFIED  (Specify)  


4.  RECOMMENDATIONS  (Check  all  that  apply) 

□  a.  SAME  EDUCATION  PROGRAMS 

□  b.  NEW  EDUCATION  PROGRAMS  (Specify). 


□  c.  SAME  ADMINISTRATIVE  CHANGE 

□  d.  NEW  ADMINISTRATIVE  CHANGE  (Specify). 


□  e.  RESTUDY  (Estimated  Mo/Yr) 

□  f.  MCE  STUDY 

□  g.  CHANGE  IN  CONCURRENT  REVIEW  (Specify). 


□  h.  OTHER  REVIEW  MODIFICATION  (Specify) 


CJi.  NO  ACTION  NECESSARY 
□  j.  OTHER  (Specify)  


5.  PERSON  HOURS  UTILIZED  (Round  to  half  hours) 

a.  PHYSICIAN  __iZ?_  

b.  OTHER  4  

c.  TOTAI  


4  1/2 


ADDITIONAL  EXPLANATORY  REMARKS      (USE  ADDITIONAL  PAPER  IF  NECESSARY) 

Inservice  training  program  von  cardiac  monitoring  was  conducted;  restudy  showed  no  delays 
in  start  of  monitoring  for  suspected  AMI  patients.    Emergency  team  was  organized  and 
began  work  in  E.R.  on  11/1/74.   Radiolog  y    dept.  instituted  new  procedures  for  X-rays 
of  E.R.  patients;  delays  in  obtaining  X-rays  have  now  been  eliminated. 
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Only  data  accumulated  by  June  1,  1977  are  included  in  this  report. 
Because  of  the  above  factors,  it  is  recognized  that  some  MCEs  done 
in  the  quarters  closest  to  this  deadline  may  be  excluded. 

Methodology 

An  accounting  was  done  of  MCE  audit  activity  completed  from 
the  geginning  of  the  program  in  1975  through  the  last  month  in  1976. 
Tabulabtions  were  made  of  the  number  of  audits  and  their  distribution 
by  PSRO  and  by  subject  content  area.    Rank  ordering  was  done  for  topic 
categories  and  their  classifications. 

c 

Although  much  of  the  data  are  presented  by  quarter,  for  clarity 
of  display,  the  data  also  were  grouped  by  quarters  into  two  basic  obser- 
vation periods. 

Observation  Period  I  -  Quarter  2,  1975  through  Quarter  J, 

1976 

Observation  Period  II  -  Quarters  2  through  4,  1976 
Additional  data  which  became  available  after  the  start  of  this  report 
are  shown  as  updated  information. 

Findings  and  Analysis 

Quantity  and  Distribution  of  MCE  Audits 

Chart  I  shows  that  total  number  of  MCE  audits  reported  through 
the  PMIS  since  early  1975.    The  number  rises  dramatically  until  the  first 
quarter  of  1977.    The  subsequent  decline  is  a  function  of  reporting  delays. 
Updated  information  for  Periods  I  and  II  is  indicated  by  the  clear  bars. 
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Chart  1 

MCE  Initial  Audit  Activity,  By  Quarter 
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Chart  II  presents  the  growth  in  MCE  audit  activity  both  in 
the  number  of  PSROs  reporting  audits  and  in  the  average  number  of  audits 
per  reporting  PSRO.    During  observation  period  I,  41  PSROs  reported  audits 
as  compared  to  a  total  of  56  of  the  original  65  PSROs  reporting  audits 
during  observation  period  II.    This  growth  in  PSRO  activity  reflects  the 
PSROs"  increased  operational  capabilities'.    There  is  a  start-up  time 
required  for  PSROs  to  develop  the  necessary  organizational  capacity  to 
perform  review  functions  as  well  as  to  delegate  any  or  all  of  those 
functions  to  hospitals.    During  its  early  stages,  the  PSRO  directs  most 
administrative  attention  towards  concurrent  review.    Consequently,  the 
implementation  of  MCEs  is  a  relatively  late  development,  generally  one 
year  after  the  institution  of  concurrent  review. 

As  shown  in  Table  I,  the  56  reporting  PSROs  display  a  wide 
variation  in  MCE  audit  activity,  ranging  from  only  1  audit  to  317  as  of 
June  1,  1977.    The  most  active  PSRO  in  this  group  is  the  Mississippi 
Foundation  for  Medical  Care,  Inc.  (PSRO  Number  MS01),  whose  audits  account 
for  10%  of  all  MCE  activity.    Further,  Chart  III  demonstrates  that  45% 
of  all  MCE  activity  can  be  attributed  to  10  PSROs.    Such  concentrations 
of  activity  naturally  limit  the  conclusions  which  can  be  drawn  from  these 
data. 

Increases  in  the  number  of  MCE  audits  reported  over  time  tend 
to  reflect  increased  productivity  of  the  10  most  active  PSROs.    This  can 
be  seen  in  Table  I,  Quarterly  Distribution  of  MCE  Audit  Activity. 
There  are,  however,  two  PSROs  which  had  minimal  activity  during  Observa- 
tion Period  I  but  are  among  the  most  active  of  Observation  Period  II. 
One  PSRO  exhibited  the  reverse  trend. 
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Chart  II 

Average  PSRO  MCE  Audit  Activity,  By  Quarter 
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Table  I 

Quarterly  Distribution  of  MCE  Audit  Activity 
(Reported  by  PSROs  as  of  June  1,  1977) 
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Chart  III; 

Percent  of  Total  MCE  Activity,  By  Ranked  Groupings 
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Compliance  with  MCE  Numerica l_Requirements 
The  numerical  requirements  for  the  performance  of  MCEs  were 
distributed  on  January  25,  1977  through  PSRO  Transmittal  No.  43.  (See 
Chapter  2,  background)    Under  binding  review,  a  minimum  of  4  MCEs  per 
year  is  required  per  hospital.    In  contrast,  the  largest  hospitals  must 
perform  12  per  year.    Since  these  requirements  are  per  calendar  year, 
a  calculation  of  compliance  with  the  regulation  cannot  be  made  earlier 
than  March  1978.    A  report  is  now  planned  for  that  time. 

In  an  effort  to  encourage  compliance,  data  on  the  numbers 
of  MCEs  done  by  each  PSRO  per  quarter  and  per  year  are  sent  to  all 
PSROs  via  PMIS  Output  Reports.    These  reports  began  in  June  1977.  With 
this  information,  an  individual  PSRO  can  compare  it's  own  activity  to 
that  of  similar  PSROs. 

MCE  Content  Areas 
Major  Classifications 

The  topic  of  each  individual  MCEs  is  reported  on  the  PMIS 
forms.    For  analysis  purposes,  topics  were  placed  by  the  contractor 
into  one  of  three  internally  consistent  classifications  which  are 
based  on  medical  care  procedures.    Judgments  about  assigment  of  audit 
topics  into  the  "medical",  "surgical",  and  "administrative  and  miscel- 
laneous procedures"  classification  were  made  independently  by  two 
trained  raters.    Inconsistencies  were  reconciled  by  senior  project 
personnel.    All  coding  decisions  were  reviewed  for  appropriateness  and 
accuracy  by  the  project  monitor,  a  physician.    The  classifications  were 
defined  as  follows: 
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•  Classification  A:  Medical 

This  classification  includes  all  medical  diagnoses 
and  procedures  for  which  the  generally  accepted 
treatment  is  medical.    Ulcers,  leukemias,  lymphomas 
and  neoplasms  not  treated  surgically  were  included 
in  this  classification. 

•  Classification  B:  Surgical 

This  classification  includes  all  surgical  and  obste- 
trical procedures  as  well  as  invasive  diagnostic 
procedures.    It  also  includes  those  diagnoses  for 
which  surgery  is  generally  indicated,  such  as  breast 
lesions  and  cholelithiasis.    Neoplasms  usually  treated 
surgically  were  included  in  this  classification. 

•  Classification  C:    Administrative  and  Miscellaneous 

Procedures 

This  classification  includes  administrative  topics 

such  as  audits  and  examination  of  length  of  stay. 

Studies  on  the  use  of  particular  facilities  or  services 

within  the  hospital,  such  as  emergency  room  and  x-rays, 
were  coded  into  Classification  C. 

Table  II  shows  the  distribution  of  audits  for  the  two 
observation  periods.    The  proportion  of  audits  in  each  classification 
remained  constant  over  time;  about  45%  were  medical,  45%  surgical,  and 
9%  administrative.    There  was  nearly  a  three-fold  increase  in  activity 
across  all  classifications  in  Period  II,  indicating  a  strong  proportion- 
ate growth.    In  Table  III,  the  same  information  is  broken  down  by 
quarters.    Quarter  2-75  stands  apart  due  to  its  small  number  of  MCE 
audits  and  somewhat  different  distribution.. 

Topic  Categories 

MCE  topics  are  selected  by  the  physicians  who  design  the 
studies,  and  may  be  highly  specific  to  a  paritcular  condition  or 
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Table  II 

Distribution  of  MCE  Audits  by  Major  Classification 


* 

Classification 

Period  I 

(41  PSROs) 

Period  II  (56  PSROs) 
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audits  1n 
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fication 

Number  of 
audits  1n 
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cation 
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fication 
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2,131 

.  100.0% 
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100.0% 
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Table  III 

Quarterly  Distribution  of  MCE  Audits  by  Major  Classification 
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for  Quarter  3 

Number  of  Audits  in 
Classification 

Percent  of  All  Audits 
in  Quarter  4 

Kank  ot  uassitication 
for  Quarter  4 

Number  of  Audits  in 
Classification 

Percent  ot  Audits 
Involved 

A 

18 

60.0 

1 

38 

46.3 

1 

108 

43.7 

2 

176 

45.6 

1 

303 

45.8 

2 

154 

44.7 

2 

302 

44. £ 

2 

1 ,299 

45.2 

B 

10 

33.3 

2 

35 

42.7 

2 

118 

47.8 

1 

173 

44.8 

2 

306 

46.3 

,  1 

363 

45.8 

1 

306 

45.1 

1 

1,311 

45.6 

C 

2 

6.7 

3 

9 

11.0 

3 

21 

8.5 

3 

37 

9.6 

3 

52 

7.9 

3 

75 

9.5 

3 

70  . 

10.3 

3 

266 

9.2 

Total 

30 

100% 

82 

100% 

247 

100% 

386 

661 

100% 

792 

100% 

578 

100°/ 

2,876 

100% 

its  treatment.    Since  a  high  degree  of  specificity  was  not  useful  for 
this  analysis,  categories  were  constructed  by  content-analyzing  the 
study  topics.    Groups  or  "topic  categories"  were  then  formed  using 
•standard  H-ICDA  descriptors.    Senior  project  personnel,  including  a 
physician,  reviewed  the  sorting  process  for  accuracy  and  appropriateness. 
The  result  was  a  reduction  of  more  than  '2,876  audit  topics  into  305 
relatively  homogeneous  topic  categories.    For  the  most  part,  these  topic 
categories  represented  conditions,  procedures  or  diagnoses  such  as 
hypertension,  delivery,  or  myocardial  infarction. 


DIAGRAM  II:  Process  of  Arranging  MCE  Topics 
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The  Twenty  Most  Frequently  Audited  Topic  Categories 


Table  IV  shows  the  twenty  topic  categories  most  frequently 
chosen  for  audit  in  each  observation  period.    The  rank  and  classifica- 
tion are  also  given.    Some  of  the  findings  are: 

1)  While  there  are  some- shifts  in  rank  among  the 
topics,  the  ranking  remains  generally  stable 
from  Period  I  to  Period  II.  The  correlation 
between  the  two  rankings  is  statistically  signi- 
ficant at  the  .01  level. 

2)  Myocardial  infarction,  pneumonia,  cholecystec-1 
tomy,  and  appendectomy  were  the  four  most  fre- 
quently audited  topic  categories  in  both  time 
periods.    However,  they  represented  4%  less  of 
the  audit  activity  in  Period  II,  possibly  indi- 
cating a  trend  toward  topic  diversification. 
This  is  not  a  statistically  significant  differ- 
ence. 

3)  The  same  trend  toward  diversification  is  possibly 
indicated  by  the  fact  that  while  the  top  twenty 
categories  accounted  for  56.7%  of  all  audits  in 
Period  I,  they  represented  only  53.9%  of  Period 
II's  activity.    That  is,  the  amount  of  audits 
done  on  topic  categories  other  than  the  twenty 
most  frequent  rose  from  43.3%  in  Period  I  to  46.1% 
in  Period  II.    These  are  not  statistically  signi- 
ficant differences. 

4)  No  single  topic  category  dominates  the  list. 
Table  V  presents  a  quarterly  breakdown  of  the  same 
information.    Within  any  quarter,  myocardial  infarc- 
tion, pneumonia,  and  cholecystectomy  consistently 
ranked  among  the  most  frequently  audited.  Topic 
categories  which  received  high  audit  attention  in 
one  or  more  quarters  without  making  the  overall 
"top  twenty"  list  include  urinary  tract  infection, 
depression,  antibiotics,  diverticulitis,  and 
thrombophlebitis. 

On  the  average,  the  twenty  most  frequently  chosen 
topic  categories  account  for  57%  of  the  audits 
performed  during  a  quarter.    This  means  that  the 
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Table  IV 

The  Twenty  Most  Frequently  Initially  Audited  Topic-Related  Category 
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Percent  of 
all  audits 

Percent  of 
all  audits 

TOTAL,  Periods  I  I  II 

1 f  1 ca- 
tion 

Period  I 
(41  PSROs) 

Period  II 
(56  PSROs) 

Period  II 
(56  PSROs) 

Period  I 
(41  PSROs) 

Period  II 
(56  PSROs) 

Number  of 
audi  ts 

Percent  of 

a  1  T   Audi t< 
ail   auui li 

A 

Myocardial  Infarction 

1 

1 

49 

106 

6.6 

5.0 

155 

5.4 

A 

Pneumonia 

2 

2 

42 

94 

5.6 

4 . 4 

136 

4.7 

B 

Cholecystectomy 

3 

3. 5 

40 

84 

5.4 

3. 9 

124 

4.3 

B 

Appendectomy 

4 

3.5 

27 

84 

3.6 

3.9 

111 

3.9 

B 

Hysterectomy 

5 

5 

25 

69 

3.4 

3.2 

94 

3.3 

B 

C-5ect1on 

7.5 

8.5 

21 

62 

2.8 

2.9 

83 

2.9 

B 

Tonsillectomy  and 
Adenomectomy 

9 

8.5 

19 

62 

2.6 

2.9 

81 

2.8 

B 

H1p  Fracture 

10 

7 

18 

63 

2.4 

3.0 

81 

2.8 

A 

Diabetes 

12.5 

6 

16 

65 

2.1 

3.1 

81 

2.8 

B 

Prostatectomy 

6 

10.5 

24 

50 

3.2 

2.3 

74 

2.6 

B 

Breast  Lesions 

15 

10.5 

15 

50  ' 

2.0 

2.3 

65 

2.3 

C 

Emergency  Room 

12.5 

12 

16 

46 

2.1 

2.2 

62 

2.2 

B 

Inguinal  Hernia 

13 

14 

16 

43 

2.1 

2.0 

59 

2.1 

A 

Gastroenteritis 

18 

13 

12 

45 

1.6 

2.1 

57 

2.0 

A 

Ulcer 

16.5 

15 

14 

42 

1.9 

2.0 

56 

1 .9 

B 

Cataract 

16.5 

16 

14 

41 

1.9 

1.9 

55 

1 .9 

A 

Hypertension 

13 

18 

16 

38 

2.1 

1 .8 

54 

1.9 

B 

Del  1  very 

7.5 

21 

2.8 

52 

1.8 

A 

Cerebrovascular 
Accident 

17 

39 

1.8 

48 

1.7 

A 

Congestive  Heart 
Failure 

20 

19 

9 

37 

1.2 

1.7 

46 

1.6 

A 

Asthma 

19 

20 

10 

32 

1.3 

1.5 

42 

1.5 

TOTAL 

424 

1,152 

56.7 

53.9 

1.616 

56.0 

Table  V 

The  Twenty  Most  Frequently  Audited  Topic  Categories  in  Each  Quarter 
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"o  ^  — 

Myocardial  Infarction 

16.7 

1.5 

5 

6. 1 

3.5 

16 

6.8 

1 

23 

6.0 

1 

37 

5.6 

1.5 

35 

4.4 

2 

34 

6.0 

1 

155 

5.4 

Pneumonia 

A 

5 

16.7 

1.5 

6 

7.3 

2 

13 

5.3 

2.5 

18 

4.7 

2.5 

37 

5.6 

1.5 

29 

3.7 

5.5 

28 

4.1 

2.5 

138 

4.7 

Cholecystectomy 

1 

1 

3.3 

6 

8 

9.8 

1 

13 

5.3 

2.5 

18 

4.7 

2.5 

23 

3.5 

6 

42 

5.3 

1 

19 

2.8 

7 

124 

4.3 

Appendectomy 

1 

- 

- 

3 

3.7 

7.5 

9 

3.6 

5.5 

14 

3.6 

S.5 

31 

4.7 

3 

29 

3.7 

5.5 

25 

3.7 

4 

111 

3.9 

Hysterectomy 

5 

- 

- 

^ 

r  5 

9 

3.6 

5.5 

12 

3.1 

7 

16 

2.4 

10 

31 

3.9 

3 

22 

3.3 

5 

94 

3.3 

C- Section 

1 

6.7 

4 

7 

2.8 

7 

10 

2.6 

8.5 

27 

4.1 

4 

18 

2.3 

11.5 

17 

2.5 

105 

81 

2.8 

Tonsil lectomy  and 
Adenomectomy 

E 

11 

4.5 

4 

8 

2.1 

13 

24 

3.6 

5 

20 

2.5 

9 

18 

2.7 

8.5 

81 

2.9 

H1p  Fractures 

1 

14 

3.6 

5.5 

18 

2.7 

8.5 

30 

3.8 

4 

15 

2.2 

12.5 

77 

2.7 

Di  abe  tes 

4 

4.9 

5.5 

| 

2.0 

1.8 

16.5 

13 

2.7- 

8. 5 

19 

2  4 

10 

4.1 

2.5 

81 

2  S 

Pr )Statectomy 

B 

- 

2 

2.4 

11 

c 

9. 5 

1 7 

4.4 

4  . 

2.0 

13. r 

24 

3*0 

7 

l.a 

17 

2.6 

Emergency  Room 

[ 

2.4 

6 

2.4 

9. 5 

2. 1 

10 

1.5 

1 7  ' 

16 

2.3 

18 

2 . 7 

S.  5 

62 

2.2 

Breast  Lesions 

1 

9 

2.3 

10 

20 

3.0 

7 

- 

- 

20 

2.9 

6 

65 

2.3 

Inguinal  Hernia 

1 

- 

- 

2 

2.4 

11 

5 

2.0 

14.5 

8 

2.1 

13 

15 

2.3 

11 

13 

1.5 

19 

15 

2.2 

12.5 

59 

2.1 

Gastroenteritis 

A 

- 

3 

3.7 

7.5 

8 

2.1 

13 

2.1 

12 

2. 1 

14.5 

2.1 

14 

2.0 

Hypertension 

A 

2 

6.7 

4 

2 

2.4 

11 

S 

2.0 

14  r 

l.E 

10 

1.5 

17 

2.1 

11 

1.6 

1' 

54 

1.9 

Cataract 

B 

6 

2.4 

9.S 

e 

2. 1 

13 

* 

' 

- 

13 

1.6 

19 

17 

2.5 

1 05 

55 

1.9 

Delivery 

5 

- 

- 

5 

6.1 

3.5 

5 

2.0 

54 

1.9 

Congestive  Heart 
Failure 

A 

4 

1.6 

19.5 

10 

1.5 

17.5 

17 

2.1 

14.5 

46 

1.6 

Cerebrovascular 
Accident 

A 

4 

1.6 

19.5 

- 

- 

- 

■ 

• 

- 

22 

2.5 

8 

- 

- 

- 

48 

1.7 

Ulcer 

A 

- 

- 

- 

2 

2.4 

11 

* 

- 

- 

6 

1.6 

18.5 

12 

1.8 

15 

16 

2.0 

17 

11 

1.5 

17 

56 

1.9 

Asthma 

A 

4 

1.6 

19.5 

- 

- 

13 

2.0 

13.5 

- 

- 

- 

10 

1.5 

19.5 

42 

1.5 

Urinary  Tract 
Infection 

A 

5 

2.0 

14.5 

- 

- 

8 

1.2 

20.5 

- 

- 

- 

- 

34 

1.2 

Depression 

A 

6 

1.6 

18.5 

■ 

17 

2.1 

14.5 

a: 

1.5 

Dilatation  and 
Curettage 

£ 

12 

1.8 

15 

32 

1.1 

Chronic  Obstructive 
Pulmonary  Disease 

A 

13 

1.6 

19 

31 

1.1 

Patient-Admissions, 

Transfers, 

Discharge 

C 

10 

l.S 

19.5 

30 

1.0 

Gastrointestinal 
Cancer 

E 

10 

1.5 

17.5 

29 

1.0 

Genitourinary 
Cancer 

E 

6 

Diverticulitis 

A 

4 

1.6 

29 

t 

1.0 

Thrombophlebitis 

8 

1.2 

20.5 

20 

0.7 

length  of  Stay 

C 

2 

6.7 

4 

7 

C.2 

low  Back  Pain 

1 

5 

2.0 

14.' 

11 

0-4 

TOTAL 

17 

567 

40 

505 

140 

59.9 

211 

54.7 

376 

56.9 

440 

55.6  | 

355 

52.3 

1 .957 

ti.2 
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remaining  43%  of  audit  activity  is  spread  over 
a  wide  range  of  topic  categories  which  consequently 
receive  little  attention.    This  set  of  less 
frequently  audited  topic  categories  may  include 
up  to  285  members. 

The  Ten  Most  Frequent  Medical  Topic  Categories 

As  indicated  by  Table  VI,  the  most  frequently  audited  topic" 
categories  in  Classification  A  (Medical)  remained  essentially  constant 
during  the  two  observation  periods.    Though  a  rank  ordering  indicated 
some  change  in  relative  frequency  of  investigation  by  topic  categories 4 
this  change  is  not  large.    The  correlation  between  the  two  rank  order- 
ings  is  statistically  significant  at  the  .01  level. 

Though  the  overall  number  of  initial  audits  in  Classification 
A  almost  tripled  between  Observation  Periods  I  and  II,  the  percentage 
of  initial  audits  in  Classification  A  represented  by  the  10  most  fre- 
quently audited  topic  categories  remained  largely  unchanged.    The  10 
most  frequently  audited  topic  categories  represented  54.5%  of  all  audits 
in  Classification  A  during  Observation  Period  I,  and  54.7%  during 
Period  II.    The  correlation  between  these  frequencies  is  statistically 
significant  at  the  .01  level. 

Myocardial  infarction  and  pneumonia  were  the  most  frequently 
audited  topic  categories  in  Classification  A  during  both  observation 
periods.    Diabetes,  gastroenteritis,  and  ulcers  were  also  frequently 
audited. 
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Table  VI 

The  Ten  Most  Frequently  Audited  Topic  Categories  in  Classification  A  (Medical) 


Ifumlier  of 
audits 

fVrlmt  1 

(41  PSItOs) 

Number  of 
audits 

1  I.I  IUII  II 

(56  PSROs) 

Percent  of 

total  In 

,  - 1  , ,  r  i  *  i 
classifi- 
cation 
Period  1 
(41  PSROs) 

rcilclli  ui 

total  In 

CluSSII  1  - 

catlon 
Period  It 
(56  PSROs) 

TOTAL.  Periods  I  &  1 1 

Topic  Cateqory 

Rank  o 
Period  I 
(41  PSROs) 

Rank 

Period  II 
(56  PSROs) 

Number  of 
audits 

Percent  of 
all  in  class- 
ification A 

Myocardial  Infarction 

1 

1 

49 

106 

14.4 

11.0 

155 

11.9 

Pneumonia 

2 

2 

42 

94 

12.4 

9.0 

136 

10.4 

Diabetes 

3.5 

3 

16 

65 

4.7 

6.8 

81 

6.2 

Gastroenteritis 

6 

4 

12 

45 

3.5 

4.7 

57 

4.4 

Ulcer 

5 

5 

14 

42 

4.1 

4.4 

56 

4.3 

Hypertension 

3.5 

7 

16 

30 

4.7 

4.0 

54 

4.1 

Cerebrovascular  Accident 

9 

6 

9 

39 

2.6 

4.1 

40 

3.7 

Congestive  Heart  Failure 

9 

0 

9 

37 

2.6 

3.9 

46 

3.5. 

Asthma 

7 

9 

10 

32 

r-  2.9 

3.3 

42 

3.2 

Depression 

9 

9 

2.6 

42 

3.2 

Urinary  Tract  Infection 

10 

25 

2.6 

34 

2.6 

TOTAL 

106 

523 

54.5 

54.7 

751 

57.5 

c 


The  Ten  Most  Frequent  Surgical  Topic  Categories 
Table  VII  shows  that  topic  categories  which  were  frequently 
audited  in  Classification  B  (Surgical)  tended  to  be  constant  across 
the  two  observation  periods.    Though  there  was  some  variation,  the 
correlation- of  rank  orderings  of  the  categories  in  the  two  observa- 
tion periods  is  statistically  significant  at  the  .05  level.  The 
fluctuation  in  the  ranking  of  topic  categories  revolved  around 
delivery,  which  was  more  frequently  audited  in  Period  I,  and  cata- 
racts, which  were  more  frequently  audited  in  Period  II. 

Among  surgical  topic  categories,  cholecystectomy,  appen- 
dectomy and  hysterectomy  were  the  most  frequently  audited.  Caesarean 
section,  tonsillectomy  and  adenoidectomy ,  and  hip  fractures  also  were 
frequently  audited. 

The  10  most  frequently  audited  surigical  topic  categories 
accounted  for  67.4%  of  the  audits  in  Classification  B  during  Period  I, 
and  61.3%  during  Period  II.    The  correlation  between  the  proportion 
of  audits  represented  by  the  10  most  frequently  audited  topic  categories 
during  Periods  I  and  II  is  not  statistically  significant.    The  decrease 
of  6.1  percentage  points  suggests  that  as  time  goes  on  the  concentra- 
tion of  audit  activity  on  the  10  most  prominent  surgical  topic  categories 
tends  to  be  redistributed,  and  that  a  greater  variety  of  surgical  sub- 
jects are  frequently  reviewed  by  the  MCE  process. 

It  may  be  noted  that  initially  the  10  most  frequently  audited 
surgical  topic  categories  accounted  for  a  greater  proportion  of  all 
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Table  VII 

The  Ten  Most  Frequently  Audited  Topic  Categories  In  Classification  B  (Surgical) 


i up i kg i a  tea  category 

Rank 

Period  I 

//il    nrnrir  \ 
\1l  rbKUSj 

Rank 

Period  II 

/rr    nr  nr\-  \ 

(56  PSROs) 

Number  of 
audi  ts 
Period  I 
(41  PSROs) 

Number  of 
audits 
Period  II 
(56  PSROs) 

Percent  of 
total  in 
classifi- 
cation 
Period  I 
(41  PSROs) 

Percent  of 
total  in 
classi  fi- 
cation 
Period  II 
(56  PSROs) 

TOTAL,  (1975  and  1976) 

Number  of 
audits 

Percent  of 
all  In  class- 
ification B 

Cholecystectomy 

1 

1.5 

40 

84 

11.9 

8.6 

124 

9.5 

Appendectomy 

2 

1.5 

27 

84 

8.0 

8.6 

111 

8.5 

Hysterectomy 

3 

3 

25 

69 

7.4 

7.1 

94 

7.2 

C-Section 

5.5 

5.5 

21 

62 

6.3 

6.4 

83 

6.3 

Tonsillectomy  &  Adenoidectomy 

7 

5.5 

19 

62 

5.7 

6.4 

Rl 

Q  •  C 

Hip  Fracture 

8 

4 

10 

63 

5.4 

6.5 

81 

6.2 

Pros  ta  tpr  tomv 

7  1 

.  0.1 

74 

5.6 

Breast  Lesions 

10 

7.5 

15 

50 

4.5 

5.1 

65 

5.0 

Inguinal  Hernia 

9 

9 

16 

43  • 

4.8 

4.4 

59 

4.5 

Cataract 

10 

41 

4.2 

55 

4.2 

Del ivery 

5.5 

21 

6.3 

52 

4.0 

TOTAL 

226 

608 

67.3 

62.4 

879 

67.2 

• 


surgical  audits  than  the  proportion  of  all  medical  audits  accounted 
for  by  the  10  most  frequently  audited  topic  categories  in  the  medical 
classification.    During  Period  I  the  top  10  topic  categories  accounted 
for  67.3%  of  surgical  audits  and  54.5%  of  medical  audits. 

As  time  goes  on,  however,  this  difference  tends  to  diminish. 
During  Period  II,  audits  of  the  top  10  surgical  topic  categories 
accounted  for  62.4%  of  all  audits  in  the  classification,  whereas  the 
top  10  medical  topic  categories  accounted  for  54.7%  of  all  audits  in 
that  classification. 

Thus,  as  time  progresses,  the  initial  concentration  of 
surgical  interest  in  the  10  most  popular  topic  categories  redistributes 
to  more  closely  reflect  the  distribution  within  the  medical  classifi- 
cation. 

The  Ten  Most  Frequent  Administrative  and  Miscellaneous 
Topic  Categories  ~ 

The  information  displayed  on  Table  VIII  demonstrates  that 
administrative  and  miscellaneous  procedures  are  the  least  frequently 
audited  of  the  MCE  topics.    The  relative  quantity  of  audits  performed 
in  this  classification  is  small,  only  9.2%  of  the  total.    The  fre- 
quency with  which  topics  in  this  classification  were  audited  is  not 
of  a  sufficient  magnitude  to  permit  a  rank  order  correlation. 

Certain  emergency  room  procedures  such  as  patient  admissions, 
transfer,  discharge  procedures,  and  antibiotics  were  the  most  frequently 
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audited  topic  categories  in  Classification  C.  Blood  usage,  trans- 
fusions, and  laboratories  were  also  frequently  investigated. 


3.2  Conclusions  from  Chapter  III 

The  analysis  done  in  this  first  section  revealed  the  fol- 


lowing: 


1.  Of  the  65  original  PSROs,  56  have  instituted 
MCE  activity  and  are  reporting  audits  throuqh 
the  PMIS. 

2.  The  total  number  of  MCE  audits  completed  each 
quarter  is  rapidly  growing.    Approximately  1,000 
audits  were  completed  each  quarter  during  observa- 
tion Period  II.    For  the  first  quarter  of  1977, 
1,342  MCE  audits  were  reported  in  the  PMIS. 

3.  The  distribution  of  reported  MCE  audits  is  con- 
centrated in  10  PSROs  whose  activity  accounts 
for  45%  of  the  total.    During  the  last  quarter 
analyzed  (4-76),  the  average  number  of  audits 
per  PSRO  is  16.1. 

4.  Since  annual  requirements  for  the  numbers  of 
MCEs  per  hospital  were  issued  effective  January 
1977,  compliance  to  that  ruling  can  be  assessed 
in  March  1978.    In  the  interim,  the  quarterly 
PMIS  Output  Report  informs  PSROs  of  their  rela- 
tive level  of  activity. 

5.  Few  MCE  topics  address  global  issues  of  medical 
care  or  patient  outcome.    Instead,  topics  tend 
to  be  specific  medical  diseases,  surgical  pro- 
cedures, or  discrete  aspects  of  patient 
administration.    Topics  are  almost  equally  dis- 
tributed within  these  first  two  classifications, 
medical  and  surgical;  each  claiming  about  45% 

of  the  total  audit  activity.  Administrative 
procedures  are  chosen  for  topics  approximately 
9%  of  the  time. 

6.  The  20  most  frequently  audited  topic  categories 
constitute  56%  of  all  MCE  audits.    There  is 
evidence  to  suggest  that  this  concentration  is 
lessening  somewhat  as  time  passes. 
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4. 


ANALYSIS  OF  MCE  RESULTS  IN  FOUR  MAJOR  TOPIC  CATEGORIES 


4.1- 


Introduction 


This  second  section  of  the  evaluation  report  presents  an 


analysis  of  the  process  and  results  obtained  from  the  assessment  of  all 
completed  MCE  audits  and  reaudits  in  the  categories  of  myocardial  infarc- 
tion, cholecystectomy,  appendectomy  and  pneumonia.    For  each  topic,  all 
deficiencies  found  in  medical  care  or  patient  outcome  are  displayed  in 
tabular  form  (Exhibits  I- IV )  along  with  their  rate  of  occurrence. 
Deficiencies  are  then  categorized  according  to  problem  areas  (Tables 
XII-XV).    An  examination  of  the  information  on  each  of  these  Exhibits 


can  be  considered  successful.    Improvements,  or  lack  of  improvements, 
in  deficiency  rates  point  out  whether  or  not  these  MCEs  have  had  at 
least  a  short  term  impact.    Four  subchapters,  one  for  each  topic  cate- 
gory, are  presented  below.    Each  of  these  subchapters  summarizes  the 
overall  adequacy  of  the  MCE  process  for  the  studies  examined  and 
summarizes  the  MCE  results  for  each  of  the  four  topic  categories  con- 
sidered.   Finally,  the  last  subchapter  examines  the  capabilities  of 
the  MCE  as  a  tool  for  revealing  and  removing  problem  areas. 


and  Tables 


indicates  the  degree  to  which  MCEs  in  the  four  categori- 


es 
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4.2  Methodology  for  the  Analysis  of  MCE  Results 

Methodology 

Based  on  their  frequency  of  completion  and  above-average 
likelihood  of  containing  high  impact  MCEs  (see  Chapter  5  for  an 
explanation  of  the  rating  system  that  determined  this  likelihood), 
the  topic  categories  myocardial  infarction,  pneumonia,  appendectomy 
and  cholecystectomy  were  "selected  for  an  assessment  of  the 
results  of  MCEs.    A  catalog  of  all  completed  and  reported  audits  and 
reaudits  was  developed  for  each  of  the  four  topic  categories.  In 
cases  where  a  full  audit/reaudit  cycle  hadobeen  made,  the  responsible 
PSROs  were  requested  to  send  to  the  contractor  all  audit  notes,  sum- 
maries and  other  data  relevant  to  these  particular  studies.    Table  VIII 
consists  of  a  distribution  by  topic  category  of  the  number  of  PSRCs 
represented  by  this  collection  of  studies,  the  number  of  reaudits 
reported  completed,  the  number  of  studies  actually  submitted  by  the 
PSROs  after  they  were  requested  to  do  so,  and  the  number  actually 
found  acceptable  for  evaluation.    From  the  MCEs  sent  in  by  various 
PSROs,  ten  were  selected  for  analysis  in  the  three  categories, 
myocardial  infarction,  pneumonia  and  appendectomy.    Seven  were 
selected  for  analysis  in  the  category  cholecystectomy. 

The  data  in  Table  IX  are  current  as  of  August  15,  1977  which 
was  three  days  beyond  the  deadline  for  submission  by  the  PSROs  of 
requested  studies.    A  review  of  the  table  shews  that  while  there  were 
not  many  reaudits  completed  in  each  topic  area,  for  each  of  the  four 
selected  topic  categories,  there  were  more  than  ten  reaudits  reported 
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Table  VIII 

The  Ten  Most  Frequently  Audited  Topic  Categories  in  Classification  C 
(Administrative  and  Miscellaneous  Procedures) 


Topic-Related  Category 

Rank 

1   LI    IUU  1 

(41  PSROs) 

Rank 

Po»-i  n/1    T  I 
rcl  1  U-U    I  l 

(56  PSROs) 

Ml  JMIDL'T     U  I 

audi  ts 

1  Li  lUtl  1 

(41  PSROs) 

iiuiiiiJL  r  u  i 
audi  Ls 

Pnrin/I  IT 

(56  PSROs) 

i erccn c  or 
total  in 
classi  f  i  - 
cation 

Pnr  1  r\f\  T 
rLi  IUU  1 

(41  PSROs) 

Percent  of 
total  in 
classi  fl- 
ea tion 

rtJl   1  (JU     1  1 

(56  PSROs) 

TOTAL,  (1975  and  1976) 

Number  of 
audits 

Percent  of 
all  1n  class- 
ification C 

Emergency  Room 

1 

1 

16 

46 

23.2 

23.4 

•fi? 

?3  3 

Patient  Admissions,  Transfer, 

2 

2.5 

0 

22 

11.6 

11.2 

30 

11.3 

Di  s charge 

Antibiotics 

4 

2.5 

5 

22 

7.2 

11.2 

27 

10.2 

Plood,  Usage  &  Transfusions 

5 

4 

4 

10 

'  5.0 

9.1 

22 

8.3 

Laboratories 

3 

5 

7 

9 

10.1 

4.6 

16 

6.0 

X-Ray 

6 

0 

4.1 

0 

3.0 

Length  of  Stay 

6 

0.5 

3 

4 

4.3 

2.0 

7 

2.6 

Intermittent  Positive 

7 

6 

3.0 

6 

2.3 

Pressure  Breathing 

Shock  Therapy 

0.5 

4 

2.0 

4 

1.5 

Physical  Therapy 

10 

3 

1.5 

3 

1.1 

TOTAL 

43 

142 

62.2 

72.1 

185 

69.6 
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Table  IX 

The  Number  of  PSROs  Reporting  Completed  Studies.   The  Number  of  Studies 
Reported  Completed,  the  number  of  Reported  Studies  Submitted  for  Eval- 
uation and  the  Number  of  Studies  Found  Acceptable  for  Evaluation 


Topic  Category 

Number  of 
PSROs  Report-, 
ing  Studies 
Completed 
Through 
Re  audit 

Number  of 

Studies 

Reported 

Completed? 

Through 

Reaudit 

'  Number  of 
Completed 
Studies  Sub- 
mitted for 
Evaluation 

Number  of 

Studies 
Found 
Accept- 
able 
For 
Evaluation 

Myocardial  Infarction 

15 

29 

25 

23 

Pneumonia 

8 

18 

12 

11 

Cholecystectomy 

11 

14 

12 

7 

Appendectomy 

8 

17 

16 

12 

completed.    While  all  of  those  reported  completed  were  not  submitted 
to  the  contractor,  more  than  ten  studies  were  submitted  for  all  the 
topic  categories.    The  reasons  the  PSRCs  had  for  not  submitting  stud-5? 
were  varied  but  primarily  fell  into  two  categories.    Either  the  study 
had  been  reported  inaccurately  and  was  not  actually  a  completed  MCE 
study  or  the  PSROs  were  unable  or  unwilling  to  encourage  the  hospitals 
which  performed  the  studies  to  submit  the  necessary  data.    Less  fre- 
quently, it  was  found  that  while  a  reaudit  had  been  completed,  the 
initial  audit  had  been  performed  prior  to  binding  review  and  the 
PSROs  did  not  have  legal  access  to  that  part  of  the  requisite  infor- 
mation. 

Although  an  adequate  number  of  studies  was  submitted,  it 
was  found  that  some  of  these  studies  were  not  acceptable  because 
requested  material  had  not  been  included  or  the  material  included 
was  not  complete  enough  to  allow  the  contractor  to  proceed  with  the 
evaluation.    In  total,  requests  for  data  were  made  of  twenty- four 
PSROs,  most  of  whom  were  quite  cooperative  in  terms  of  locating  and 
forwarding  data  and  in  terms  of  answering  questions  about  the  material 
submitted. 

Once  the  contractor,  received  the  material  necessary  for 
completing  the  evaluation  of  a  particular  study,  notation  of  this  fact 
was  made  and  the  PSROs  were  sent  a  letter  thanking  them  and  their 
hospitals  for  the  cooperation  they  extended  during  the  study. 
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Data  Processing  and  Analysis 

At  the  time  that  a  study  was  submitted  to  the  contractor, 
a  research  assisstant  and  a  senior  analyst  reviewed  all  the  material 
available  relevant  to  the  study  and  either  logged  it  in  as  ready  for 
analysis  or  prepared  a  list  of  questions  about  the  material  and/or 

0 

additional  materials  needed.    Questions  and  requests  for  additional 
material  were  handled  directly  with  the  PSROs  by  telephone.  These 
contacts  were  made  either  by  project  staff  or,  in  situations  where  the 
PSROs  were  less  than  cooperative,  by  the  project  officer. 

A  file  was  prepared  for  each  study  which  included  all  avail- 
able materials  relating  to  the  complete  study  and  all  the  necessary 
forms  for  abstracting  and  scoring  the  data.    Files  were  then  sent  to  a 
five-person  team  which  included  a  physician,  a  publich  health  specialist, 
a  senior  analyst  and  two  specially  trained  research  assistants.  Senior 
members  of  the  team  reviewed  and  analyzed  the  data  and  gave  each  study 
an  impact  score  based  on  a  system  to  be  described.    At  least  two  senior 
team  members  and  occasionally  three  reviewed  each  study.    Where  disagree- 
ments about  scoring  occurred  consensus  was  reached  after  careful  dis- 
cussion and  re-review  of  the  material.    Scores,  therefore  reflect  agree- 
ment among  senior  staff  from  the  fields  of  medical  care  and  medical  care 
evaluation.    According  to  prescribed  procedures,  the  junior  team  members 
abstracted  data  for  later  collation  and  final  analysis.    The  results  of. 
the  impact  scoring  and  data  analysis  appear  in  subsequent  sections  of  this 
report. 
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The  abstracting,  analysis  and  scoring  of  the  studies  was 
done  according  to  three  systems  developed  by  the  contractor  under  the 
terms  of  previous  contracts  and  for  the  purpose  of  this  evaluation. 

The  impact  score  given  each  study  consisted  of  the  combina- 
tion of  a  first  level  analysis  score  based  on  an  assessment  of  satis- 
factory completion  of  the  elements  of  the  MCE  audit  process,  and  an 'impact 
score  based  on:  the  percentage  of  types  of  documented  deficiencies 
removed  by  the  time  of  reaudit;  the  percentage  of  recommended  actions 
implemented  by  the  time  of  reaudit;  and  the  weighted  differential  poten- 
tial impact  of  types  of  corrected  deficiencies. 

The  elements  of  the  impact  score  were  weighted  so  that  primary 
emphasis  was  placed  on  the  removal  of  types  of  deficiencies  in  the  medical 
care  process  or  patient  outcomes  which  were  felt  to  have  potential  for 
major  impact  on  patients'  health  status.    Complete  descriptions  of  the 
scoring  systems  are  contained  in  Appendix  A,  First  Level  Analysis  of  Cri- 
tical MCE  Elements.    Appendix  B,  Development  and  Specification  of  the 
Method  of  Application  of  An  Impact  Rating  System  for  MCE  Studies  is  also 
included  at  the  end  of  the  report.    The  forms  used  for  abstracting  and 
scoring  data  are  included  in  Appendix  C. 

It  is  suggested  that  the  reader  review  these  materials.  However, 
in  terms  of  the  subsequent  material  describing  the  evaluation  of  each  of 
the  four  topic  categories,  it  is  important  to  note  that  the  final  impact 
scores  were  based  on  the  distribution  of  a  possible  40  points  described 
in  Table  II. 
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As  can  be  seen  from  a  review  of  Table  X,  it  is  possible 
for  a  study  to  receive  a  score  as  high  as  40.    However,  studies  were 
considered  acceptable,  that  is  essentially  well  executed  and  demonstrating 
acceptable  impact,  if  they  received  a  score  of  at  least  29.5.    One  way 
in  which  studies  could  obtain  this  minimally  acceptable  total  score  was 
if  they  received  credit  for  at  least  70%  of  the  requisite  process  factors, 
75%  for  implementing  recommendations  and  75%  of  the  combined  correction 
of  all  types  of  deficiencies  and  correction  of  types  of  major  deficiencies. 

However,  an  acceptable  score  could  also  be  obtained  if  either 
or  both  the  process  factor  score  or  the  score  for  implementing  recommen- 
dations  was  somewhat  below  the  acceptable  level.    Of  either  or  both  the 
process  factor  score  or  the  score  for  implementing  recommendations  was 
below  an  acceptable  level,  in  order  to  obtain  an  overall  acceptable  score, 
more  than  75%  of  the  types  of  deficiencies  found  would  have  to  have  been 
corrected.    It  is  felt  that  the  correction  of  deficient  medical  care 
process  or  patient  outcomes  is  by  far  the  most  important  contribution 
of  the  MCE  study.    Therefore,  even  though  process  elements  might  not  be 
quite  adequate,  a  study  could  be  considered  acceptable  if  its  results, 
that  is  the  change  in  medical  care  process  or  patient  outcomes,  were 
adequate. 

When  the  scoring  process  was  complete,  data  was  abstracted 
from  the  study  materials  and  from  the  scoring  analysis  forms.    This  data 
provided,  the  basis  for  the  anecdotal  summaries  of  studies  of  the  four 


-67- 


Table  x 


Distribution  of  40  Points  by  Scoring  Factors, 
Maximum  Possible  Score,  and  Acceptable  Score 


Scon'  ng 
Factor 

Max  i~un 

Possible 

Score 

Acceptable  . 
Score 

MCE  Process  Factor 

10 

7 

Implementation  of 
recommended  actions 

10 

7.5 

.  Correcti  on  of  types 
of  deficiencies 

10 

Differential  impact 
of  types  of  correc- 
ted deficiencies 

10 

15* 

Total 

40 

29.  5 

 i 

*  Fifteen  points   is  considered  an  acceptable  score  for 
the  combination  of  scores   for  correction  of  types  of 
deficiencies  and  score  for  differential   impact  of  types 
of  corrected  deficiencies. 
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topic  categories  which  appear  subsequently  in  this  report.    A  copy  of  the 
form  used  to  abstract  the  data  is  included  as  Appendix  C. 

Data  Validity 

A  number  of  problems  with  the  data  were  encountered  during  the 
collection  and  analysis  of  studies.    These  problems  fall  into  two  major 
categories;  those  problems  relating  to  availability  of  data  and  those 
relating  to  the  specificity  and  quality  of  that  data  which  was  available. 

In  terms  of  availability  of  data,  the  most  significant  problem, 
as  noted  earlier,  was  the  paucity  of  reaudits  which  had  been  reported  com- 

o 

pleted  by  the  PSROs.    A  small  telephone  survey  indicated  that  the  PSROs 
were  behind  in  reporting  or  were  reporting  in  a  manner  too  inaccurate 
to  be  accepted  by  the  PMIS  processors.    This  indicates  that,  in  fact, 
a  greater  level  of  reaudit  activity  is  going  on  than  is  apparent  from 
review  of  PMIS  data  outputs.    The  magnitude  of  this  differential  in  actual 
and  reported  activity  is  unknown. 

There  are  two  other  factors  which  are  felt  to  contribute  signi- 
ficantly to  the  low  level  of  reaudit  activity.    The  first  factor  relates 
to  the  lack  of  incentives  for  reaudit  built  into  current  MCE  guidelines. 
A  hospital  is  not  given  credit  for  an  MCE  study  if  the  need  for  reaudit 
was  identified,  until  the  reaudit  is  completed.    The  full  cycle  may  take 
more  than  1  year  to  complete.    Also,  reaudits  are  not  required  unless  the 
hospital  decides  the  deficiencies  are  sufficiently  important  to  warrant 
reaudit.    It  is  easier  for  a  hopsital  to  complete  an  audit  to  indicate 
that  no  reaudit  is  necessary  and  obtain  credit  quickly  than  for  the  hosDital 
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to  do  a  reaudit.    The  second  factor  contributing  to  the  low  level  of 
reaudit  activity  relates  to  the  fact  that  MCE  review  is  always  the  last 
type  of  review  to  be  implemented  after  a  PSRO  becomes  conditional.  Con- 
sequently, for  many  PSROs,  not  enough  time  has  elapsed  for  much  reaudit 
activity  to  have  taken  place.    These  factors  are  discussed  more  fully 
in  earlier  sections  of  this  report. 

In  addition  to  the  relatively  small  numbers  of  completed  MCE 
studies  available,  it  was  difficult  to  obtain  sufficient  data  about  audits 
on  which  to  base  an  evaluation  decision  for  those  studies  which  were  com- 
pleted.   In  some  cases,  materials  were  reported  "missing"  or  "unavailable" 
or  data  were  so  incomplete  as  to  be  meaningless.    The  PSROs  in  many  instances 
were  wary  of  "stressing"  their  relationships  with  their  hospitals  by  making 
"unusual"  requests  for  data.    In  addition,  even  though  all  identifiers  .were 
to  be  deleted,  there  was  concern  about  confidentially  especially  related 
to  audits  which  included  non-federal ly  subsidized  patients  or  to  initial 
audits  completed  before  th  initiation  of  binding  review. 

While  all  of  the  above  problems  with  the  data  were  encountered 
to  a  greater  or  lesser  extent,  it  was  possible,  through  interpolation  and 
by  employing  liberal  scoring  standards,  to  use  much  of  the  material  sub- 
mitted.   Some  of  the  studies  submitted  were  complete,  excellent  in  detail 
and  apparently  effective  in  bringing  about  significant  positive  change 
in  deficient  medical  care  process  and  patient  outcomes.    These  studies 
could  have  received  acceptable  scores  even  with  the  use  of  much  more 
stringent  standards  than  those  used  for  the  present  study.    Most  of  the 
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studies  evaluated  indicate,  at  the  very  least,  a  good  potential  for 
effecting  the  type  of  change  just  described.    Even  those  studies  that 
required  more  liberal  standards  to  pass,  suffered  mostly  from  inadequacy 
of  documentation  of  the  audit  process,  rather  than  from  the  medical  care 
itself.    With  better  documentation  on  the  part  of  the  hospitals  and  PSROs 
and  with  more  time  for  the  contractor  to  extract,  or  clarify  and  validate 
all  of  the  available  data,  it  is  likely  that  more  improvement  of  medical 
care  process  and  of  patient  outcomes  could  have  been  demonstrated.  Thus, 
it  is  possible  to  say  that  with  all  the  above  mentioned  limitations,  the 
data  nevertheless  indicate  that  the  MCE  study  has  potential  as  a  tool  for 
effecting  positive  change  in  deficient  medical  care  process  and  patient 
outcomes . 

4.3  Results  from  MCEs  Performed  in  the  Topic  Category: 

•  MYOCARDIAL  INFARCTION 

Myocardial  Infarction:    Study  Process 

In  terms  of  process  factors  relating  to  the  implementation  of 
MCE  studies  on  myocardial  infarction,  the  following  characteristics  are 
of  note. 

Validity  and  Specificity  of  Topic  Selection: 

Myocardial  infarction  was  ranked  eighth  of  the  twenty  most  fre- 
quently audited  MCE  topic  categories  for  its  probability  of  elucidating 
removable  problem  areas. -   Half  of  the  studies  had  reasonably  specific 
topic  titles  and  objectives.    The  situation  specific  reasons  for  selecting 
the  topic  ranged  from  high  incidence  and  prevalence  (6  studies)  to  perceived 
need  for  evaluation  of  the  diagnosis  and  management  procedures  (1  study)  and 

1/  See  Chapter  b,  Table  VI.  " 
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evaluation  of  the  use  of  the  cardiac  care  unit  (1  study).    Few  studies 
contained  indications  of  the  method  used  for  selecting  the  study  topic. 

Given  these  facts,  it  is  possible  to  say  that  myocardial  infarc- 
tion is  a  validly  selected  topic  in  that  audits  correcting  deficient  prac- 
tice or  outcome  have  significant  potential  for  positive  impact  on  patient 
health  status.    In  addition,  there  were  good  situation  specific  reasons 
for  selecting  the  topic  in  the  hospitals  whose  audits  were  under  considera- 
tion. 

Locus  of  Study  Responsibility  and  Personnel  Involvement: 
Hospital  medical  audit  committees  were  responsible  for  5  of  the 
studies,  while  PSRO  Regional  Audit  Committees  were  responsible  for  the 
other  5  studies.    Five  initial  audits  were  conducted  on  a  region-wide 
basis  and  5  were  conducted  by  individual  hospitals.    All  reaudits  were 
conducted  by  individual  hosptials. 

Six  of  the  studies  submitted  contained  no  documentation  of  non- 
physician  participation,  3  studies  had  no  non-physician  participation  and 
one  had  a  RRA  Health  Record  Analyst  involved.  For  all  of  the  audits  and 
reaudits  submitted,  a  range  of  1  to  120  non-physician  hours  were  utilized 
with  a  median  of  17  1/2  hours  and  a  mean  of  27  hours  being  utilized.  The 
number  of  physician  hours  utilized  ranged  from  1  to  45  with  a  median  of 
5  hours  and  a  mean  of  9  hours  utilized. 

Audit  Format  and  Criteria  Selection: 

All  of  the  studies  were  based  on  a  retrospective  collection  of 
data.    Five  of  the  studies  used  the  standard  JCAH/PEP  audit  format  while 


5  of  the  studies  used  special  PSRO  audit  formats.    Six  of  the  studies  used 
regionally  generated  criteria,  2  of  the  studies  employed  self-generated 
criteria  and  2  of  the  studies  used  criteria  generated  by  national  organi- 
zations.   The  number  of  criteria  elements  employed  for  audits  and  reaudits 
ranged  from  6  to  33  with  a  mean  of  15  and  a  median  of  13  criteria  utilized 
Six  of  the  reaudits  used  all  of  the  criteria  while  4  of  the  reaudits 
employed  a  subset  'of  criteria  using  only  those  for  which  deficiencies 
had  been  documented  during  the  initial  audit.    Written  lists  of  criteria 
elements  were  submitted  for  all  studies  evaluated  and  written  or  verbal 
lists  of  standards  were  submitted  for  all  studies  evaluated. 

Sampling  Characteristics: 

The  procedures  used  for  sampling  charts  varied  from  "the  selec- 
tion of  the  last  50  cases"  to  selection  of  "all  cases  with  a  diagnosis  of 
myocardial  infarction"  for  periods  of  time  ranging  from  2  months  to  14 
months.    The  sample  of  charts  for  most  of  the  audits  and  reaudits  were 
consecutive  cases  selected  over  a  period  of  4  to  6  months. 

The  sample  size  for  audits  and  reaudits  also  varied  quite  con- 
siderably from  20  charts  to  72  charts,  with  a  median  of  41  charts  and 
a  mean  of  39  charts.    Documentation  indicated  that  all  studies  included 
patients  of  both  sexes  and  of  all  ages.    Specific  age  ranges  were  not 
indicated.    Additionally,  in  terms  of  sampling  characteristics,  the 
audits  and  reaudits  varied  in  the  number  of  physicians  audited  from  4  to 
38  with  a  median  of  7  and  a  mean  of  11.3  physicians  audited. 
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Myocardial  Infarction:  Study  Results 

As  noted  earlier  in  this  report,  this  evaluation  was  designed 
to  look  at  the  results  of  MCE  studies  as  well  as  at  the  process  of  their 
implementation.    The  final  impact  scores  include  both  process  and  result 
factors.    The  results  of  the  sample  of  myocardial  infarction  studies  will 
be  aggregated  for  discussion. 

Table  XI    consists  of  a  tabulation  of  the  overall  process  and 
impact  scores  which  were  given  to  10  of  the  25  myocardial  infarction 
studies  evaluated  during  this  study.    The  10  studies  selected  for  discus- 
sion here  consist  of  those  studies  which  were  evaluated  as  having  been 
implemented  in  the  most  rigorous  way  and  which  appeared  to  have  brought 
about  the  most  substantial  amounts  of  change. 

As  Table  XII  indicates,  although  one  of  the  studies  received 
a  score  for  elimination  of  deficiencies  which  was  slightly  below  acceptable 
(14.8  vs.  15),  all  the  studies  being  considered  here  were  given  overall 
scores  above  the  threshold  level  of  acceptability  (29.5  points).  The 
final  impact  scores  ranged  from  34.8  to  40.    Documentation  that  recommended 
actions  had  been  implemented  was  absent  from  almost  all  material  received 
for  all  studies  considered  during  this  evaluation.    However,  careful  review 
of  audit  notes  and  correspondence  frequently  allowed  the  assumption  to  be 
made  that  the  actions  had  been  implemented.    Therefore,  it  was  decided  that 
because,  at  this  point  in  time,  the  hospitals  uniformly  appear  not  to  be 
meeting  the  requirements  (22.8)  that  actual  implementation  of  recommended 
actions  be  documented,  and  because  the  assumption  often  could  be  made 
that  actions  had  been  implemented,  the  full  10  points  would  be  given 
every  study  evaluated. 
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As  review  of  Exhibit  I  shows,  an  analysis  of  deficiencies  indi- 
cated that  the  10  studies  highlighted  15  separate  categories  of  deficiencies. 
Of  these  categories,  8  related  to  deficiencies  in  process  of  a  clinical  nature, 
6  related  to  deficiencies  in  process  having  to  do  with  adequate  or  appro- 
priate documentation  and  1  related  to  a  deficiency  in  the  area  of  patient 

care  outcomes.    Of  the  15  categories  of  deficiencies  5,  or  33%,  were  found 
to  occur  in  more  than  1  study.    Of  these  5  categories  of  deficiencies, 

1  was  found  in  7  of  the  studies,  2  in  6  of  the  studies,  1  in  5  of  the 

studies,  and  1  in  3  of  the  studies.    The  most  frequently  occurring 

category  of  deficiencies  was'  that  of  failure  to  admit  patients  to  the  CCU 

while  the  next  most  frequently  occuring  categories  of  deficiencies  were 

failure  to  electronically  monitor  patients,  the  occrrence  of  unjustified 

mortality,  and  failure  to  maintain  I.V.  lines.    Of  the  total  number  of 

categories  of  deficiencies,  9  or  60%  were  considered  to  be  major.  In 

other  words,  occurrence  of  these  deficiencies  had  potential  for  directly 

negatively  affecting  patients'  outcomes  for  the  given  episode  of  illness 

or  perhaps  ultimately  in  terms  of  their  health  status. 

The  extent  (percentage  of  non-compliance  with  standard)  to  which 

categories  of  deficiencies  were  elucidated  during  the  initial  audit  varied 

from  2%  to  100%.    The  extent  to  which  the  same  deficiencies  were  found 

during  the  reaudit  varied  from  0%  to  28%.    The  change  in  extent  of  deficient 

process  and  outcomes  of  medical  care  represented  by  the  decrease  in 

deficiency  from  initial  to  reaudit  is  substantial.    The  percentage  change 

varied  frcm  0%  to  100%  with  an  overall  mean  change  of  86%,  a  consistently 

high  level  of  change.    As  noted  earlier,  the  level  of  change  was  considered 


Table  XI.     Distribution  of  Process  and  Impact  Scores 
For  Ten  Myocardial  Infarction  Studies 


Total  Score  (Matched 
Process  and  Impact  Scores) 

Acceptable 
Study  Score 

TOTAL 

Proc. 

Imple. 
of  Rec. 

Elim.  of 
Def. 

YES 

NO 

40 

10 

10 

20 

+ 

40 

10 

10 

20 

0 

+ 

39.5 

8.5 

10 

20 

+ 

39 

9 

10 

20 

+ 

39 

9 

10 

20 

+ 

39 

9 

10 

20 

+ 

38.5 

8.5 

10 

20 

+ 

37.5 

7.5 

10 

20 

+ 

37.2 

10 

10 

17.2 

+ 

34.8 

10 

10 

14.8 

+ 

Total 
Studies 
Graded:  10 
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Table    XII:  Frequency  Distribution  of  Primary 
Types  of  Recommended  Action 


Type  of  Action 

Frequency  of  Recommendation 

Practitioner  counseling 

18 

Educational  programs . 

11 

Staff  meeting 

6 

Letter  to  medical  staff 

3 

Monitoring  of  specific 
physicians'  charts 

2 

Revision  of  criteria 

2 

Administrative  change 

1 

TOTAL 

43 
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Exhibit  1:    Dis tribunes-:  of  Deficiencies  Found  in 
Ten  Myocardial  Infarction  Studies 


Category  of 
Deficiency 

Overall  Classi- 
fication of 
Deficiency 

Designation 
of  Deficiency 
major:  M  minor:  m 

Number  of 
Studies  in 
Which  Defici- 
ency Occurred 

Percent  Extent 

of  Unjustified 
Deficiency  ir. 
Initial  Audit 
Ranee      1  Mean 

Percent  Extent 
of  Unjustified 
Deficiency  in 
Re-Audit 
Range     1  Mean 

Percent 
Change  in 
Deficiency 

Acceptable 
Level  of 
Change 

pc  ;  po  i  ?co 

Range !  Mean 

Yes     1  Ho 

Failure  to 
admit  pati- 
ent to  ecu 

x  !  ! 

M 

7 

2-1        !  16. 1 

0-17     !  4.8 

84-100'  96.1 

x  ! 

Late  or  no 
electronic 
monitoring 
of  patient 

x  !  ! 

M 

• 

6 

12.1-50  !  22.9 

0-14      1  3.6 

72-100;  90. S 

x  ! 

Failure  to 
document 
patient  on 
monitor 
within  30 
minutes 

!  x  ! 

M 

1 

24         !  24 

8           ]  8 

67      !  67 

x  j 

Failure  to 
maintain 
I.V.  line 

!  x  I 

H 

5 

14.5  [24.2 

0-10       1  3.5 

50-100  166.4 

x  ! 

Failure  to 
document 
patient  on 
I.U.  with- 
in 30 
minutes 

j  | 
i  x  ! 

M 

1 

j 

26          |  26 

j 

7           !  7 

73      j  73 

| 

X 

Failure  to 
obtain  EKG 
on  admission 

x  j  j 

m 

1 

6         |  6 

0           ;  0 

100  [100 

X 

Errors  in 
EKG  inter- 
pretation 

x  ' 

M 

1 

20         !  20 

0           |  0 

100  lioo 

X 

PVC's  but 
no  Rx 

x  !  ! 

M 

1 

8         !  8 

0          !  0 

100  !ioo 

x  ! 

Failure  to 
repeat  enz- 
yme studies 
in  instances 
of  elevated 

d  Mi  ytuc  o 

 1 — !  

j  [ 

x  !  ! 

M 

1 

2  2 

 !  

I 

o     !  o 

ioo   J  ioo 

j 

x  ! 

Failure 
to  obtain 
urine 
analysis 

x  | 

M 

1 

10  10 

8       j  8 

20     |  20 

x  j 

No  chest 
X-ray 
report  in 
charts 

j    X  | 

m 

1 

10         (  10 

0      ]  0 

100     ]  100 

x  I 

No  EKG 
during  last 
week  docu- 
menting 
improvement 

x  1 

m 

1 

2  2 

0  0 

100    |  100 

x  ! 

Failure  to 
document 
patient 
ambulatory 

]  i 
i  *  j 

m 

1 

2          i  23 

12.5     i  12.5 

46     j  46 

i 

x  j 

Inadequate 
documenta- 
tion of 
discharge 
instructions 

i  x  ! 

m 

3 

10-36       !  28.6 
 -1  

16-28   1  20.6 

0-53%  jl7.6 

x(l)  J  x(2) 

Mortality 

!    i  '  x 

M 

6 

15-25      |  22 

0-8     j  1.3 

67-100i94.5 

X  ! 

*  PC:  Process/Clinical 

PO:  Process /Documentation 
PCO:    Patient  Care  Outcome 
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acceptable  in  90%  of  the  studies.    In  no  case  was  a  negative  change  or 
increase  in  deficiency  documented  in  the  reaudit. 

Review  of  the  studies  indicated  that  for  every  category,  the 
deficiencies  in  medical  care  process  and  outcome  were  attributed  to  groups 
of  physicians.    In  a  few  situations,  particularly  those  relating  to 

problems  of  documentation  and  availability  of  CCU  beds,  deficiencies 
also  were  attributed  to  hospital  administration. 

Seven  major  types  of  educational  and/or  remedial  interventions 
were  recommended  as  a  result  of  findings  during  the  initial  audit.  Table 
consists  of  a  frequency  distribution  of  these  recommended  actions. 

For  all  the  instances  of  deficient  medical  care  process  or  out- 
come in  all  of  the  studies  evaluated,  43  recommendations  for  educational 
interventions  or  remedial  actions  were  made.    Sixty-seven  percent  of  these 
recommendations  involved  two  types  of  action,  practitioner  counseling  and 
educational  programs.    Fifty-three  percent  of  the  actions  were  ones  appli- 
cable to  all  physicians  involved  in  the  treatment  of  myocardial  infarction, 
while  47%  of  the  actions  were  applicable  to  those  physicians  directly 
responsible  for  the  deficient  practice  or  outcomes. 

In  summary,  the  results  of  an  evaluation  of  10  MCE  studies  on 
myocardial  infarction  are  indicative  of  two  basic  points.    All  the  studies 
evaluated  (highest  scores  were  selected)  received  acceptable  process  scores 
and  90%  received  acceptable  impact  scores  indicating  that,  given  the  fairly 
liberal  standards  for  grading,  it  appears  that  these  studies  were  conducted 
in  a  reasonably  rigorous  manner  and  resulted  in  sufficient  change,  i.e., 
reduction,  in  deficient  medical  care  process  or  outcome. 

The  results  of  initial  myocardial,  infarction  audits  and  the 
proximate  results  or  positive  change  demonstrated  by  reaudit  which  have 
been  discussed  in  this  section  underscore  the  potential  that  the  MCE  study 


has  for  revealing  removable  problem  areas  relating  to  medical  care  pro- 
cess and  outcome.    Large  portions  of  these  problems  areas  have  been 
ameliorated,  as  is  evidenced  by  the  correction  of  more  than  half  of  the 
discovered  deficiencies. 

4.4  Results  from  MCEs  Performed  in  the  Topic  Category:  PNEUMONIA 

Pneumonia:  Study  Process 

In  terms  of  process  factors  relating  to  the  implementation  of 
MCE  studies  on  pneumonia,  the  following  characteristics  are  of  note. 
Validity  and  Specificity  of  Topic  Selections: 

Pneumonia  was  ranked  fourth  of  the  twenty  most  frequently  audited 
MCE  topic  categories  for  its  probability  of  elucidating  removable  problem 
areas.-''    Four  of  the  pneumonia  studies  had  reasonably  specific  topic  titles 
and  objectives,  while  the  situation  specific  reasons  for  selecting  the  topic 
ranged  from  incidence/prevalence  (3  studies)  to  need  to  complete  an  audit 
(1  study)  and  preventable  morbidity/mortality  (1  study).    Given  these 
facts,  it  is  possible  to  say  that  pneumonia  is  a  validly  selected  topic 
in  that  audits  correcting  deficient  practice  or  outcomes  have  significant 
potential  for  positive  impact  on  patient  health  status.    In  addition, 
there  were  acceptable  situation  specific  reasons  for  selecting  the  topic 
in  the  hospitals  whose  audits  were  under  consideration. 

Locus  of  Study  Responsibility  and  Personnel  Involvement: 
Hospital  medical  audit  committees  were  responsible  for  3  of  the 
studies,  the  medical  service  in  hospitals  was  responsible  for  2  studies 
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and  5  studies  were  conducted  by  PSROs  as  part  of  regional  audits. 

Two  of  the  studies  submitted  contained  documentation  that  there 
had  been  no  non-physician  participation.  5  studies  had  ?SRC  personnel 
involved,  2  studies  had  nurses  involved,  1  study  had  a  medical  audit 
assistant  involved  and  1  study  had  other  medical  reocrds  personnel  involved. 
For  4  audits  and  reaudits  which  contained  data  about  person  hours  utilized, 
there  was  a  range  of  2  to  33  1/2  non-physician  hours  reported  utilized  with 
a  mean  of  19.3  hours  and  a  median  of  24  hours  being  utilized.    The  number 
of  physician  hours  for  audit  and  reaudits  ranged  from  1/2  to  30  hours  with 
a  mean  of  3.3  hours  and  a  median  of  5.5  hours  being  utilized. 

The  sample  size  for  audits  and  reaudits  also  varied  quite  con- 
siderably from  10  charts  to  50  charts,  with  a  mean  of  23.5  charts  and  a 
median  of  22.5  charts.    Documentation  indicated  two  studies  were  of  pedia- 
tric populations,  7  were  of  all  ages  and  one  other  had  "no  accompanying  age 
range  data. 

Audit  Format  and  Criteria  Selection: 

All  of  the  studies  were  based  on  retrospective  collection  of 
data.    Four  of  the  studies  used  the  standard  JCAH/PEP  Audit  format,  while 
5  of  the  studies  used  PSRC  audit  formats  and  1  stucy  used  a  derivation 
of  a  nationally  used  format.    Five  of  the  studies  used  regionally  generated 
criteria,  3  of  the  studies  employed  self-generated  criteria,  1  of  the 
studies  used  criteria  generated  by  another  PSRO  and  the  remaining  study 
used  criteria  generated  by  a  national  organization.    The  number  of  cri- 
teria elements  employed  for  audits  and  reaudits  ranged  from  3  to  23,  with 
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a  mean  of  11.8  and  a  median  of  11.    Four  of  the  reaudits  used  all  of 
the  criteria,  while  6  of  the  reaudits  employed  a  subset  of  criteria 
using  only  those  for  which  deficiencies  had  been  documented  during  the 
initial  audit.    Written  lists  of  criteria  elements  were  submitted  for 
all  studies  evaluated  and  written  or  verbal  lists  of  standards  were 
submitted  for  all  studies  evaluated. 

Sampling  Characteristics: 

The  procedures  used  for  sampling  charts  varied  from  the  "selec- 
tion of  the  last  50  cases"  to  "selection  of  all  cases  with  a  diagnosis 
of  pneumonia"  for  periods  of  time  ranging  from  1  month  to  1  year.  The 
sample  of  charts  for  most  of  the  audits  and  reaudits  consisted  of  conse- 
cutive cases  selected  over  a  period  of  1  month  to  1  year.    The  sample  size 
for  audits  and  reaudits  also  varied  considerably  from  10  charts  to  50  charts, 
with  a  median  of  38  and  a  mean  of  34.    Documentation  indicated  that  studies 
included  patients  of  both  sexes  and  all  ages.    Additionally,  in  terms  of 
sampling  characteristics   the  audits  and  reaudits  varied  in  the  number  of 
of  physicians  audited  from  2  to  14  with  a  median  of  6  and  a  mean  of  5.4 
physicians  audited. 

Pneumonia:  Study  Results 

As  noted  earlier  in  this  report,  this  evaluation  was  designed 
to  look  at  the  results  of  MCE  studies  as  well  as  at  the  process  of  their 
implementation.    The  final  impact  scores  include  both  process  and  result 
factors.    The  results  of  the  sample  of  pneumonia  studies  will  be  aggregated 
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in  this  section  for  discussion. 

Table  XIII  consists  of  a  tabulation  of  the  overall  process  and 
impact  scores  which  were  given  to  10  of  the  11  studies  evaluated  during 
this  study.    The  10  studies  selected  for  discussion  here  consist  of 
those  studies  which  were  evaluated  as  having  been  implemented  in  the 
most  rigorous  way  and  which  appeared  to  have  brought  about  the  most 
substantial  amounts  of  change.    As  Table  XIII  indicates,  only  four  of 
the  studies  being  considered  here  were  given  scores  above  the  threshold 
level  of  acceptability  (29.5  points).    The  final  impact  scores  ranged 
from  19  to  40  with  a  mean  score  of  30.1  and  a  median  of  27.8.    The  pro- 
cess scores  contributing  to  these  scores  ranged  from  8  to  10  with  a  mean 
score  of  9  and  a  median  score  of  9.2.    The  score  for  implementation  of 
recommendations  was  automatically  10  points  and  the  range  of  scores  for 
elimination  of  types  of  deficiencies  ranged  from  0  to  20  points,  with  a 
mean  score  of  11.1  points  and  a  median  score  of  9.5  points.    Six  studies 
received  scores  for  elimination  of  deficiencies  which  were  below  threshold 
(15  points)  and  in  these  cases  the  variation  from  threshold  was  from  5 
to  15  points.    In  two  studies,  there  was  no  documentation  at  all  of  a 
satisfactory  decrease  in  the  level  of  deficiency. 

A  summary  review  of  Table  XIII  indicates  that  only  40%  of  those 
studies  considered  here  eliminated  acceptable  levels  of  types  of  deficienc 
ies  for  all  the  criteria  which  were  found  deficient  during  the  initial 
audit  and  which  were  reaudited.    As  review  of  Exhibit  II  shows,  an  analysi 
of  deficiencies  indicates  that  the  10  studies  highlighted  12  separate 
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Table  XIII:  Distribution  of  Process  and  Impact 
Scores  for  Ten  Pneumonia  Studies 


Total  Score  (Matched 
Process  and  Impact  Socres) 

Acceptabl e 
Study  Score 

TOTAL 

Proc. 

Impl e. 
of  Rec. 

Elim.  of 
Oef 

YES 

NO 

40 

10 

'10 

20 

+ 

40 

10 

10 

20 

+ 

40 

10 

10 

.20 

+ 

39 

9 

10 

20 

+ 

28.3 

8.8 

10 

10 

26.3 

8.5 

10 

8.3 

24 

7 

10 

7 

23.3 

8 

10 

5.3 

— 

20 

10 

10 

0 

19 

9 

10 

0 

Total' 
Studies 
Graded:  10 
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Exhibit  II:    Distribution  of  Deficiencies  in 
Ten  MCE  Pneumonia  Studies 


Category  of 
Deficiency 

Ove 
fic 
Def 
PC 

rail 

atior 

icien 

PI' 

I — — 

Of 

cv 

POO 

Designation 
of  Deficiency 
Major:  M  . 
Minor:  m 

Number  of 
Studies  in 
Which  Defici- 
ency Occurred 

Percent  Extent 
of  Unjustified 
Deficiency  in 
Initial  Audit 
Ranee  Mean 

Percent  Extent 
of  Unjustified 
Deficiency  in 
Re-Audit 
Range  Msa.^ 

Percent 
Change  in 
Defici- 
ency 

Ranee  .^ear; 

Accept 
Level 
Changs 
Yes 

able 
of 

No 

No  positive  X- 
ray  diagnosis 

X 

M 

3 

6-20  10.3 

0  0 

100  100 

+ 

No  positive 
X-ray  plus 
respiratory 
distress  of 
inability  to 
code  out  of  ' 
hospital  as 
indicators 
for  admis- 
sion 

X 

M 

1 

27  27 

No  data 

No  data 

No 
data 

No  positive 
culture  for 
pathogens 

X 

M 

7 

28-02.6  49.2 

0.71  30.7 

-34 

-+100  32.4 

X 

XC4) 

No  blood 
pressure  and 
weight  taken 
on  admission 

X 

m 

1 

19  19 

24  24 

-20  -20 

No  documen- 
tation of 
clinical 
improvement 

X 

m 

3 

14-50  32-6 

0-30.6  17.9 

-21 

-+82  33 

XC1) 

X(2) 

No  documenta- 
tion of  clini- 
cal impression 
of  pneumonia 

X 

m 

1 

37.5  37.5 

0  0 

100  100 

X 

Antibiolic 
therapy  in 
absence  of 
bacterial 

investiga- 
tion 

X 

M 

4 

37.5-44  40.4 

27.64  48.2 

-55+30  -12.5 

X 

Repeated  X- 
ray  demonstra- 
ting improve- 
ment 

X 

M 

2 

40-57  49 

18-24  21 

40-68  54 

XC1) 

X(l) 

Length  of 
stay  outside 
of  norms 

X 

m 

1 

4  4 

0  0 

0  0 

X 

Patients  not 
afebrile  for 
48  hours  prior 
to  discharge 

X 

M 

1 

8  8 

0  0 

0  0 

X 

No  resolved 
or  clearing 
X-ray  at 
discharge 

x 

M 

1 

48  48 

0  0 

0  0 

X 

No  documen- 
tation of 
follow-up 
at  discharge 

X 

m 

4 

12-81.3  38.8 

0-23.1  10 

42+100  32.6 

XC3) 

X(l) 
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categories  of  deficiencies.    Of  these  categories,  9  related  to  deficienc- 
ies in  process  of  a  clinical  nature,  while  3  related  to  deficiencies  in 
process  having  to  do  with  adequate  or  appropriate  documentation.  Unjusti- 
fied mortality  or  other  deficiencies  related  to  patient  care  outcomes 
were,  not  disclosed  by  these  audits.    Of  the  12  categories  of  deficiency, 
6,  or  50%  were  found  to  occur  in  more  than  1  study.    Of  these  6  categories 
of  deficiency  1  was  found  in  7  of  the  studies,  2  in  4  of  the  studies, 
2  in  3  of  the  studies  and  1  in  2  of  the  studies.    The  most  frequently 
occurring  category  of  deficiencies  was  that  of  failure  to  obtain  positive 
sputum  cultures  while  the  next  most  frequently  occurring  categories  of 
deficiencies  were  failure  to  document  plan  of  follow-up,  the  use  of 
antibiotic  therapy  in  the  absence  of  bacterial  investigation,  failure 
to  document  clinical  improvement  in  patients,  no  positive  X-ray  diagnosis 
and  failure  to  obtain  repeated  X-rays  to  demonstrate  clinical  improvement. 
Of  the  total  number  of  categories  of  deficiencies,  6,  or  50%  were  considered 
to  be  major.    In  other  words,  occurrence  of  these  deficiencies  had  poten- 
tial for  negatively  affecting  patients'  outcomes  for  the  given  episode 
of  illness  or  perhaps  ultimately  in  terms  of  their  health  status. 

The  extent  (percentage  of  non-compliance  with  standard)  to 
which  categories  of  deficiencies  were  revealed  during  the  initial  audit 
varied  from  4%  to  81.3%.    The  extent  to  which  the  same  deficiencies  were 
found  during  the  reaudit  varied  from  0%  to  71%.    The  change  in  extent 
of  deficient  process  or  outcomes  of  medical  care  represented  by  the  decrease 
in  deficiency  from  initial  to  reaudit  is  substantial.    The  percentage 
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change  varied  from  -56%  (negative  change)  to  +100%.    As  noted  earlier, 
the  level  of  change  was  considered  acceptable  in  only  40%  of  the  studies. 
In  6  situations,  negative  changes  or  increases  in  deficiency  were  docu- 
mented in  the  reaudit. 

Review  of  the  studies  indicated  that  for  every  category,  the 
deficiencies  in  medical  care  process  and  outcome  were  attributed  to  groups 
of  physicians . 

Five  major  types  of  educational  or  remedial  interventions  were 
recommended  as  a  result  of  findings  during  the  initial  audit.  Table  XIV 
consists  of  a  frequency  distribution  of  thes.e  recommended  actions. 


Table  XIV:  Frequency  Distribution  of  Primary 
Types  of  Recommended  Action 


Type  of  Action 

Frequency  of  Recommendation 

Letter  to  medical  staff 

19 

Practitioner  counseling 

5 

Staff  meeting 

i 

Revision  of  criteria 

1 

Administrative  change 

1 

TOTAL 

27 
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For  all  instances  of  deficient  medical  care  process  or  out- 
come in  all  of  the  studies  evaluated,  27  recommendations  for  educational 
interventions  or  remedial  actions  were  made.    Seventy  percent  of  these 
recommendations  involved  one  type  of  action,  letters  sent  to  medical 
staff  regarding  the  problems.    Eighty-one  percent  of  the  actions  were 
ones  applicable  to  all  physicians  involved  in  the*  treatment  of  pneumonia 
while  19%  of  the  actions  were  applicable  to  those  physicians  directly 
responsible  for  the  deficient  practice  or  outcomes. 

In  summary,  the  results  of  an  evaluation  of  10  MCE  studies 
on  pneumonia  are  indicative  of  two  basic  points.    Only  4  of  the  studies 
evaluated  (highest  scores  were  selected)  received  acceptable  process 
and  impact  scores  indicating  that,  given  the  fairly  liberal  standards 
for  grading,  it  appears  that  while  these  studies  were  conducted  in  a 
reasonably  rigorous  manner,  this  did  not  result  in  sufficient  change, 
i.e.,  reduction  in  deficient  medical  care  process  or  outcomes  in  more 
than  one  half  of  the  situations. 

The  results  of  initial  pneumonia  audits  and  the  proximate 
results  underscore  the  potential  the  MCE  study  has  for  disclosing  removable 
problem  areas  relating  to  medical  care  process  and  outcomes.    The  studies 
evaluated  here  also  show  potential  for  producing  positive  change  by 
eliminating  deficiencies. 
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4.5  Results  from  MCEs  Performed  in  the  Topic  Category: 

appendectomy; 

In  terms  of  process  factors  relating  to  the  implementation 
of  MCE  studies  on  appendectomy,  the  following  characteristics  are  of 
note. 

Validity  and  Specificity  of  Topic  Selection: 

Appendectomy  was  ranked  first  of  the  twenty  most  frequently 

audited  MCE  topic  categories  for  is  probability  of  elucidating  removable 

problem  areas.  1/ 

Three  of  the  ten  studies  had  reasonably  specific  topic  titles 
and  objectives.  The  situation  specific  reasons  for  selecting  the  topic 
ranged  from  high  incidence  and  prevalence  (6  studies)  to  perceived  need 
for  evaluation  of  the  performance  and  mangement  of  appendectomy  (1  study), 
general  interest  in  the  topic  (6  studies)  and  finally,  concern  over  the 
frequency  of  removal  of  normal  appendices  (1  study).  Few  studies  con- 
tained indications  of  the  method  used  for  selecting  the  study  topic. 

Given  these  facts,  it  is  possible  to  say  that  appendectomy 
is  a  validly  selected  topic  in  that  audits  correcting  deficient  prac- 
tice or  outcome  have  significant  potential  for  positive  impact  on 
patient  health  status.    In  addition,  there  were  good  situation  specific 
reasons  for  selecting  the  topic  in  the  hospitals  whose  audits  were 
under  consideration. 

Locus  of  Study  Responsibility  and  Personnel  Involvement 
PSRO  Regi  onal  Audit  Committees  were  responsible  for  7  of 
the  studies,  Medical  Audit  Committees  were  responsible  for  2  of  the 

Tj  See  Chapter  5,  Table  xvm 
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studies  and  finally,  a  department  of  surgery  was  responsible  for  1  of 
the  studies.    All  10  of  the  initial  audits  were  conducted  by  individual 
hospitals.    This  was  also  true  for  reaudits. 

Four  of  the  studies  submitted  contained  no  documentation 
of  non-physician  participation,  and  6  had  a  Medical  Audit  Assistant 
involved.    For  all  of  the  studies'  including  audits  and  reaudits  which 
submitted  information  on  person  hours,  a  range  of  2  to  30  non- physi- 
cian hours  were  utilized  with  a  median  of  8  hours  and  a  mean  11.5 
hours  being  utilized.    The  number  of  physician  hours  for  audits  and 
reaudits  range  from  1  to  7  with  a  median  of  2  and  a  mean  of  2.6 
hours  utilized. 

Audit  Format  and  Criteria  Selection 

All  of  the  studies  were  based  on  a  retrospective  collec- 
tion of  data.    Four  of  the  studies  used  the  standard  JCAH/PEP  audit 
format  while  6  of  the  studies  used  special  PSRO  audit  formats.  Seven 
of  the  studies  used  criteria  generated  by  PSRO  Regional  Audit  Com- 
mittees, 2  of  the  studies  employed  self-generated  criteria  and  1 
of  the  studies  did  not  submit  information  as  to  how  the  criteria 
were  generated..    The  number  of  criteria  elements  for  both  audits 
and  reaudits  ranged  from  1  to  25  with  a  median  of  1.0  and  a  mean 
of  8  utilized.    Two  of  the  reaudits  used  all  of  the  criteria,  and 
8  used  a  subset  of  criteria  using  only  those  criteria  for  which 
deficiencies  had  been  documented  during  the  initial  audit.  Written 
lists  of  criteria  elements  and  standards  were  submitted  for  all 
studies  evaluated. 
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Sampling  Characteristics 

The  procedures  used  for  sampling  charts  for  audits  and 
reaudits  varied  from  the  "selection  of  the  last  50  or  100  cases" 
to  "selection  of  all  appendectomy  cases"  for  periods  of  time  rang- 
ing from  6  months  to  2  years. 

The  sample  size  for  audits  and  reaudits  also  varied  con- 
siderably from  2  charts  to  100  charts  with  a  median  of  48  charts  and 
a  mean  of  43.    Documentation  indicated  that  all  studies  included 
patients  of  both  sexes  and  all  ages.    Specific  age  ranges  were  not 
indicated.    Additionally,  in  terms  of  sampling  characteristics,  the 
studies,  including  the  audit  and  reaudit  varied  in  the  number  of  physi- 
cians audited  from  1  to  16  with  a  median  of  2.5  and  a  mean  of  5.6 
physicians  audited. 

Appendectomy:  Study  Results 

As  noted  earlier  in  this  report,  this  evaluation  was  designed 
to  look  at  the  results  of  MCE  studies  as  well  as  at  the  process  of  their 
implementation.    The  final  impact  scores  include  both  process  and 
result  factors.    The  results  of  the  sample  of  appendectomy  studies  will 
be  aggregated  in  this  section  for  discussion. 

Table  XV  consists  of  a  tabulation  of  the  overall  process 
and  impact  scores  which  were  given  to  10  of  the  12  appendectomy  studies 
evaluated  during  this  study.    The  10  studies  selected  for  discussion 
here  consist  of  those  studies  which  were  evaluated  as  having. been 
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implemented  in  the  most  rigorous  way  and  which  appeared  to  have 
brought  about  the  most  substantial  amounts  of  change  of  all  the 
studies  evaluated  in  this  topic  area.    As  Table    XV  indicates, 
eight  of  the  studies  being  considered  here  were  given  scores  above 
the  threshold  level  of  acceptability  (29.5  points).    The  final  pro- 
cess scores  ranged  from  5.5  to  9  with  a  mean  score  of  7.7  and  a- 
median  score  of  8.    The  score  for  implementation  of  recommendations  was 
automatically  10  points  and  the  range  of  scores  for  elimination  of 
types  of  deficiencies  ranged  from  6.6  to  20  points,  with  a  mean  score 
of  17.7  and  a  median  score  of  20  points.    Only  two  studies  received 
a  score  for  elimination  of  deficiencies  which  was  below  threshold 
(15  points).    A  summary  view  of  Table  XV  indicates  that  only  80%  of  those 
studies  considered  here  eliminated  an  acceptable  level  of  types  of  defi- 
ciencies found  during  initial  audits  for  all  the  criteria  which  v/ere 
reaudited.    The  types  of  deficiencies  reaudited  and  documented  as 
eliminated  will  be  discussed  in  following  paragraphs.    While  only 
studies  receiving  high  impact  scores  are  considered  here,  the  fact  of 
note  is  that  review  of  those  studies  indicates  that  MCE  study  is  a  valu- 
able tool  for  correcting  areas  of  deficient  medical  process  and  patient 
outcomes  even  when  the  study  process  itself  is  somehwat  less  than  ideal. 

As  review  of  Exhibit  III  shows,  an  analysis  of  deficiencies 
indicates  that  the  10  studies  highlighted  9  separate  categories  of  defi- 
ciencies.   Of  these  categories  5  related  to  deficiencies  in  process  of 
clinical  nature,  2  related  to  deficiencies  in  process  having  to  do  with 
adequate  or  appropriate  documentation  and  2  related  to  deficiencies  in 


Table  XV  :    Distribution  of  Process  and  Impact  Scores 
For  Ten  Appendectomy  Studies 


Total  Score  (Matched 
Process  and  Impact  Score 

5 

Acceptable  Study 
Score 

TOTAL 

Proc, 

Imple. 
of  Rec. 

Elim. 

Or  UeT. 

YES 

NO 

39 

9 

10 

20 

+ 

39 

9 

10 

•  '  20 

X 

39 

9 

10 

20 

+ 

38 

8 

10 

20 

+ 

38 

8 

10 

20 

+ 

38 

8 

10 

20 

+ 

37 

7 

10 

20 

+ 

36 

6 

10 

20 

+ 

27.5 

7.5 

•  10 

10 

22.1 

5.5 

10 

6 . 5 
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Exhibit  III:    Distribution  of  Deficiencies  Found 
in  Ten  Appendectomy  Studies 


Category  of  Deficiency 

Overall  Classi- 
fication of 
Deficiency 

Designa- 
tion of 
Deficiency 

Number  of 
Studies  in 
Which  Defi 

of  Unjustified 
-  Deficiency  in 
initial  Audit 

Percent  Extent 
of  Unjustifiec 
Deficiency  in 
Re -audit 

Percent 
Change  in 
Deficiency 

Acceptable 
Level  of 
Change 

PC|  PD 

PCO 

1  

Major:  M 
Minor:  m 

ciency 
Occurred 

Did  not  meet  criteria 
for  justified  admission 

X  i 

m 

1 

Range  j  Mean 
2  2 

0 

0 

Rangej  Mean 
100  i  100 

Yes  JNo 
+ 

Surgery  not  justified  by 
Sufficient  signs  and 
symptoms 

X 

M 

3 

1.5-16'  10 

0-1.5 

2.8 

70-lOOj  80.6 

+  (3)j 

Gangrenous  or  perforated 
appetullx 

!  x 

m 

1 

4     !  4 

o  i 

0 

100  !  100 

+  j 

Routine  Chest  X-ray  not 
done 

x  ! 

M 

1 

1.5     J  1.5 

o  ! 

0 

100  !  100 

+  J 

Not  on  regular  diet 
with  normal  bowel 
function  at 
discharge 

x  ! 

M 

1 

! 

53     '  53 

o  ! 

0 

j 

ioo  !  ioo 

+  ■ 

Discharge  documen- 
tation deficiency: 
full  diet 

!  x  ' 

m 

1 

21  21 

o  ! 

0 

100  j.  100 

+  i 

Patient  did  not  receive 
home  care  orders 

x  1 

M 

1 

ioo    !  ioo 

20  j 

20 

80  j  80 

+  j 

Failure  to  document 
wound  status  at 
discharge 

!  x  ! 

m 

8 

.2-70    !    37. 4 

0-50 

16.6 

20-100J  85 

+  (6)j  -(2) 

Length  of  stay 

x  ! 

m 

1 

1     i  1 
i 

o  ! 

0 

100  !  100 

+  j 

the  area  of  patient  care  outcomes.    Of  the  9  categories  of  deficiencies 
2,  or  22%  were  found  to  occur  in  more  than  1  study.    Of  these  2  categor- 
ies of  deficiencies,  1  was  found  in  8  of  the  studies,  and  1  in  3  of 
the  studies.    The  most  frequently  occurring  category  of  deficiency  was 
that  of  failure  to  document  wound  status  at  discharge  while  the  next 
most  frequently  occurring  category  of  deficiency  was  surgery  not  justi- 
fied by  sufficient  signs  and  symptoms.    Of  the  total  number  of. cate- 
gories of  deficiencies,  4  or  44%  were  considered  to  be  major.  In 
other  words,  occurrence  of  those  deficiencies  had  potential  for  directly 
negatively  affecting  patients'  outcomes  for  the  given  episode  of 
illness  or  perhaps  ultimately  in  terms  of  their  health  status. 

The  extent  (percentage  of  non-compliance  with  standard)  to 
which  categories  of  deficiencies  were  elucidated  during  the  initial 
audit  varied  from  1%  to  100%.    The  extent  to  which  the  same  deficienc- 
ies were  found  during  the  reaudit  varied  from  0%  to  50%.    The  change 
in  extent  of  deficient  process  and  outcomes  of  medical  care  represented 
by  the  decrease  in  deficiency  from  initial  audit  to  reaudit  is  substantial 
The  percentage  change  varied  from  20%  to  100%  with  an  overall  mean  change 
of  84%,  a  consistently  high  level  of  change.    As  noted  earlier,  the 
level  of  change  was  considered  acceptable  in  80%  of  the  studies.  In 
no  case  was  a  negative  change  or  increase  in  deficiency  documented  in 
the  reaudit. 

Review  of  the  studies  indicated  that  for  every  category, 
the  deficiencies  in  medical  care  process  and  outcomes  were  attributed 
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either  to  groups  of  physicians  or  an  individual  physician.    In  a  few 
situations,  particularly  those  relating  to  problems  of  documentation  , 
deficiencies  also  were  attributed  to  hospital  administration  or  nurses. 

Five  major  types  of  remedial  intervention  were  recommended 
as  a  result  of  the  findings  during  the  initial  audit.    Table  XVI  con- 
sists of  a  frequency  distribution  of  these  recommended  actions. 

For  all  the  instances  of  deficient  medical  care  process  or 
outcomes  in  all  of  the  studies  evaluated,  24  recommendations  for 

remdial  actions  were  made.    Twenty-nine  percent  of  these  recommenda- 
tions involved  practitioner  counseling.    All  of  the  actions  were  ones 
applicable  to  all  physicians  involved  in  the  treatment  of  appendectomy. 

In  summary,  the  results  of  an  evaluation  of  10  MCE  studies 
on  appendectomy  are  indicative  of  two  basic  points.    Eight  of  the 
studies  evaluated  (highest  scored  were  selected)  received  acceptable 
process  and  impact  roles  suggesting  that,  given  the  fairly  liberal 
standards  for  grading,  it  appears  that  these  studies  were  conducted  in 
a  reasonably  rigorous  manner  and  resulted  in  sufficient  change,  i.e., 
reduction,  in  deficient  medical  care  process  or  outcomes. 

The  results  of  initial  appendectomy  audits  and  the  proximate 
results  or  positive  change  deomonstrated  by  reaudit  which  have  been 
discussed  in  this  section  underscore  the  potential  that  the  MCE  study 
has  for  disclosing  removable  problem  areas  relating  to  medical  care 
process  and  outcome.    These  studies  also  showed  the  ability  to  ameliora 
these  problem  areas. 
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Table  .XVI:     Frequency  Distribution  of  Primary 
Types  of  Recommended  Action 


Type  of  Action 

Frequency  of  Recommendation 

Practitioner  Counseling 

7 

Revision  of  Criteria 

6 

•  Staff  Meeting 

5 

Administrative  Change 

3 

Monitoring  of  Charts 

3 

TCTAL 

  ,  

24 
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4.6  Results  from  MCEs  performed  in  the  Topic  Category: 

CHOLECYSTECTOMY 

Cholecystectomy:  Study  Process 

In  terms  of  process  factors  relating  to  the  implementation 
of  MCE  studies  on  cholecystectomy,  the  following  characteristics  are 
of  note. 

Validity  and  Specificity  of  Topic  Selection 
Cholecystectomy  was  ranked  third  of  the  twenty  most  fre- 
quently audited  MCE  topic  categories  for  its  probability  of  elucidat- 

1/ 

ing  removable  problem  areas. 

Three  of  the  7  cholecystectomy  studies  analyzed  had  reason- 
ably specific  topic  titles  and  objectives;  the  situation  specific  rea- 
sons for  selecting  the  topic  ranged  from  incidence/prevalence  (5 
studies)  to  preventable  morbidity/mortality  (1  study).    The  method  used 
for  selecting  the  topic  of  2  of  the  cholecystectomy  studies  was 
profile  analysis.    For  2  other  studies  the  method  was  analysis  of 
medicial  records  and  for  an  additional  study  "physician  interest" 
was  listed  for  method  of  selection. 

Given  these  facts,  it  is  possible  to  say  that  cholecystec- 
tomy was  a  validly  selected  topic  in  that  audits  correcting  deficient 
practice  or  outcome  have  significant  potential  for  positive  impact 
on  patient  health  status.    In  addition  there  were  acceptable  situation 

J7    Bee  Chapter  5,  Table  xVfll 
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specific  reasons  for  selecting  the  topic  in  the  hospitals  whose  audits 
were  under  consideration. 


Locus  of  Study  Responsibility  and  Personnel  Invovlement 

Hospital  medical  audit  committees  were  responsible  for  3  of 
the  studies,  the  surgical  service  in  hospitals  was  responsible  for  2 
studies,  the  UR  Committee  for  the  6th  and  the  hospital  medical  staff 
for  the  remaining  study. 

Two  of  the  studies  submitted  contained  documentation  that 
there  had  been  no  non-physician  participation;  2  had  medical  records 
personnel  involved.    However,  for  the  6  studies  including  audit  and 
reaudit  which  contained  data  about  person  hours  utilized,  there  was 
a  range  of  2  to  48  non-physician  hours  reported  utilized  with  a  mean 
of  19  hours  and  a  median  of  19  hours  being  utilized.    The  number  of 
physician  hours  ranged  from  1  to  13  hours  with  a  mean  of  5.5  hours  and 
median  of  5.25  hours  being  utilized  for  the  audits  and  reaudits  involved 
in  the  six  studies  for  which  person  hours  were  reported. 

The  sample  size  for  both  audits  and  reaudits  also  varied  quite 
considerably  from  18  charts  to  82  charts,  with  a  mean  of  35.5  charts 
and  a  median  of  30.5  charts.    Documentation  indicated  all  studies  had 
populations  covering  all  ages  and  both  sexes. 

Audit  Format  and  Criteria  Selection: 

All  of  the  studies  were  based  on  retrospective  collection  of 
data.    Four  of  the  studies  used  the  standard  JCAH/PEP  Audit  format, 


-99- 


while  2  of  the  studies  used  PSRO  audit  formats  and  1  study  used 
a  derivation  of  a  nationally  used  format.    Two  of  the  studies  employed 
self-generated  criteria,  2  used  criteria  from  other  hospitals,  1  of 
the  studies  used  criteria  generated  by  another  PSRO  and  another  used 
criteria  generated  by  a  national  organization.    The  number  of  criteria 
elements  employed  for  both  audits  and  reaudits  ranged  from  1  to  27,  with 
a  mean  of  13.6  and  a  median  of  13.0."    Four  of  the  reaudits  used  all  of  the 
criteria,  while  3  of  the  reaudits  employed  a  subset  of  criteria  using 
only  those  for  which  deficiencies  had  been  documented  during  the  inital 
audit.    Written  lists  of  criteria  elements  were  submitted  for  all  studies 
evaluated  and  written  or  verbal  lists  of  standards  were  submitted  for  all 
studies  evaluated. 

Sampling  Characteristics 

The  procedures  used  for  sampling  charts  for  both  audit  and 
reaudit  varied  from  the  "selection  of  the  last  50  cases"  to  "selection 
of  all  cases  with  a  diagnosis  of  cholecystectomy"  for  periods  of  time  rang- 
ing from  6  months  to  1  year.    In  6  of  the  studies,  the  sample  of  charts 
for  these  audits  and  reaudits  included  consecutive  cases  selected  over 
a  period  of  6  months  to  1  year.    In  the  seventh  case,  a  random  sample  of 
50  charts  was  taken  out  of  a  total  population  of  87,  over  one  year's  time. 

Cholecystectomy:  Study  Results 

As  noted  earlier  in  this  report,  this  evaluation  was  designed 
to  look  at  the  results  of  MCE  studies  as  well  as  at  the  process  of  their 
implementation.    The  final  impact  scores  include  both  process  and  result 
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factors.    The  results  of  the  sample  of  cholecystectomy  studies  were 
aggregated  in  this,  section  for  discussion  of  the  overall  results. 

Table  XVII  consists  of  a  tabulation  of  the  overall  process  and 

impact  scores  which  were  given  to  the  seven  cholecystectomy  studies 
evaluated.    Originally  it  had  been  the  intent  to  review  a  greater  num- 
ber of  studies  and  to  select  those  which  had  been  implemented  in  the  most 
rigorous  way  and  which  appeared  to  have  brought  about  the  most  substan- 
tial amounts  of  change.    However,  data  for  only  seven  studies  (including 
reaudits)  on  this  topic  category  could  be  obtained.    It  was,  therefore, 
decided  to  include  all  seven  in  the  analysis  to  provide  a  maximum  data 
base.    As  Table  XVII  indicates,  only  four  of  the  studies  being  considered 

here  were  given  scores  above  the  threshold  level  of  acceptability  (29.5 
points).    The  final  impact  scores  ranged  from  18  to  39.5  with  a  mean 
score  of  32  and  a  median  of  33.2.    The  process  scores  contributing  to 
these  scores  ranged  from  7.5  to  9.5  with  a  mean  score  of  8.4  and  a  median 
score  of  8.    The  score  for  implementation  of  recommendations  was  auto- 
matically 10  points  and  the  range  of  scores  for  elimination  of  types  of 
deficiencies  ranged  from  0  to  20  points,  with  a  mean  score  of  13.5  points 
and  a  median  score  of  14.7  points.    Four  studies  received  scores  for 
elimination- of  deficiencies  which  were  below  threshold  (15  points)  and 
in  these  cases  the  variation  from  threshold  was  from  0.3  to  15  points. 
In  one  study,  there  was  no  decrease  in  the  level  of  the  single  deficiency 
which  had  been  identified  in  the  original,  audit. 

A  summary  review  of  Table  XVII  indicates  that  57%  of  those  studies 
considered  here  eliminated  or  made  acceptable  progress  in  eliminating 


-101- 


Table  XVII:    Distribution  of  Process  and  Impact  Scores 
for  Seven  Cholecystectomy  Studies 


Total  Score  (Matched 
Process  and  Impact  Scores) 

Acceptable 
Study  Score 

TOTAL 

Proc. 

Impl e . 
of  Rec. 

L  Mm  ot 
Def. 

Yes 

No 

39.5 

9.5 

10 

20 

+ 

39.5 

9.5 

10 

20 

+ 

38 

8 

10 

20 

+ 

33.2 

8.5 

10 

14.7 

+ 

28 

8 

10 

10 

27.5 

7.5 

10 

10 

18 

8 

10 

0 

Total 
Studies 
Graded:  7 
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deficiencies  for  all  the  criteria  which  were  reaudited.    As  review  of 
Exhibit  IV  shows,  an  analysis  of  deficiencies  indicates  that  the  7  stud- 
ies elucidated  10  categories  of  problems.    Of  these  categories,  5 
related  to  deficiencies  in  process  of  a  clinical  nature,  while  2 
related  to  deficiencies  in  processes  having  to  do  with  adequate  or  appro- 
priate documentation;  3  related  to  deficiencies  involving  patient  care 
outcomes.    Of  the  10  categories  of  deficiency  3,  or  30%  were  found  to 
occur  in  more  than  1  study.    Of  these  3  categories  of  deficiency  1  was 
found  in  3  of  the  studies  and  2  in  2  of  the  studies.    The  most  frequently 
occurring  category  of  deficiencies  was  the  failure  to  perform,  or  at  least 
to  record,  pre-operative  cholecystograms  to  confirm  diagnosis  prior  to 
surgery.      The  other  two  deficiencies  recorded  in  more  than  one  study 
were  failure  to  document  satisfactory  wound  healing  and  failure  to 
perform  EKG's  on  indicated  populations  (patients  over  forty).    Of  the 
total  number  of  categories  of  deficiencies  9,  or  90%,  were  considered 
to  be  major.    In  other  words,  occurrence  of  these  deficiencies  had 
potential  for  directly  negatively  affecting  patients'  outcomes  for 
the  given  episode  of  illness  or  perhaps  ultimately  in  terms  of  their 
health  status. 

The  extent  (percentage  of  non-compliance  with  standard)  to 
which  categories  of  deficiencies  were  elucidated  during  the  initial 
audit  varied  from  4%  to  34%.    The  extent  to  which  the  same  deficiencies 
were  found  during  the  reaudit  varied  from  0%  to  25%.    The  change' 
in  extent  of  deficient  process  or  outcomes  of  medical  care  represented 
by  the  decrease  in  deficiency  from  initial  to  reaudits  is  substantial. 
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Exhibit  IV:    Distribution  of  Deficiencies  in  Seven 
MCE  Studies  on  Cholecystectomy  Studies 


Category  of 
Deficiency 

Overall  Classi- 
fication of 
Deficiency 

Designation  of 
Deficiency 
Major:  M 
Minor:  m 

Number  of 
Studies  in 
wmcn  uenci- 
ency  Occurred 

Percent  of 
Unjustified 
Deficiency  in 
Initial  Audit 

Percent  of 
Unjustified 
Deficiency 
in  Re-audit 

Percent 
Change  in 
Defici- 
ency 

Acceptable 
Level  of 
Change 

rC 

PD     j  PCO 

Yes      ;  No 

Range  J  Mean 

Range  j Mean 

Range [Mean 

i*j  record  or  pre- 
operative cholecys- 
togram  confirming 
diagnosis  prior  to 
operation 

X 

M 

3 

9.314  i  20.3 

3-25  1  17. 3 

sJ  3'  D  /    i  £  U  .  * 

|X(2) 

No  documentation 
that  patient  was 
tolerating  diet 

i    x  i 

m 

1 

4      1  4 

4     j  4 

0     |  0 

!  x 

No  positive  Path 
Report 

I                   1  V 

I             j  X 

M 

1 

14      j  14 

8     j  8 

43     j  43 

X 

vUove  j 
90%) 

No  satisfactory 
wound  healing 

!  x 

M 

2 

22-33J  27.5 

o    ;  0 

100  jioo 

X(2)  j 

Wound  Dehi science 

!       !  x 

M 

1 

12      j  12 

0     j  0 

100  jlOO 

X 

EKG's  indicated  but 
not  done 

x  ! 

M 

2 

1  U  —9  9 1       1  ft 

U-lz  J  U 

15-100  J57. 5 

X(l)  |X(1) 

Progress  notes 
inadequate  to 
juoLiiy  variations 

x  ! 

M 

1 

8      j  8 

0     j  0 

100  [100 

X 

Electrolyte  Studies 
indicated  but  not 
done 

x  ! 

M 

1 

6      j  6 

o    i  0 

100  [100 

X 

Fasting  Blood 
Sugar  levels 
abnormal 

x  ! 

M 

1 

3      |  3 

9     j  9 

-300  j300 

j  X 

Non-performance  of 

ope  rat  i  ve  Cholonglography 

X  j 

M 

1 

4  4 
.  ,  i  

0  0 

100  lioo 

j 

X 

PC:    Process/a  inical 
PD:  Process/Documentation 
PCO:    Patient  Care  Outcome 


The  percentage  change  varied  from  -300%  (negative  change)  to  +100%. 
As  noted  earlier,  the  level  of  change  was  considered  acceptable  in 
only  57%  of  the  studies.    In  2  situations,  negative  changes  or 
increases  in  deficiency  for  individual  criteria  were  documented  in 
the  reaudit. 

Review  of  the  studies  indicated  that  for  all  but  one  cate- 
gory, the  lack  of  documentation  that" patient  was  tolerating  normal 
diet,  the  deficiencies  in  medical  care  process  and  outcomes  were  attri 
buted  to  groups  of  physicians.    The  category  of  deficiency  relating 
to  documentation  of  diet  was  attributed  to  nurses-. 

Six  major  types  of  educational  or  remedial  interventions 
were  recommended  as  a  result  of  findings  during  the  initial  audit. 
Tab! eXVm  consists  of  a  frequency  distribution  of  the  recommended 
actions. 


Table  XVni 

Frequency  Distribution  of  Primary  Types  of 
Recommended  Action 


Type  of  Action 

rreauency  or  Recommendation 

Letter  to  medical' staff 

4 

Discussion  by  Audit  Committe 
with  medical  staff 

4 

Practitioner  counseling 

4 

Letters  to  individual  physi- 
cians 

2 

Staff  meeting,  physicians 

1 

Staff  meeting,  nurses 

1 

TOTAL 

16 
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For  all  instances  of  deficient  medical  care  process  or 
outcomes  in  all  of  the  studies  evaluated,  16  recommendations  for 
educational  interventions  or  remedial  actions  were  made.  Thirty 
seven  percent  of  these  recommendations  involved  one  type  of  action: 
staff  meetings  at  which  deficiencies  were  brought  to  the  attention 
of  practitioners  as  a  group  either  at  regular  medical  staff  meetings 
where  the  audit  committee  reported  findings  and  deficiencies  or  at 
medical  or  nursing  committee  meetings  where  the  audit  results  were 
reviewed.    In  4  cases  (25%),  letters  were  sent  to  medical  staff  to 
inform  them  of  the  specific  problem(s).    In  6  cases  (37%)  remedial 
action  was  handled  through  individual  counseling  or  letters  to  the 
practitioners  involved. 

In  summary,  an  evaluation  of  7  MCE  studies  on  cholecystec- 
tomy reveals  two  basic  points:  all  received  acceptable  process  score: 
and  4  of  the  7  received  acceptable  impact  scores.    This  indicates, 
even  with  the  liberal  standards  for  grading,  that  these  studies  were 
conducted  in  a  reasonably  rigorous  manner  and  that  over  half  the 
studies  resulted  in  acceptable  action  and  reduction  in  deficient 
medical  care  or  outcome. 

The  results  of  initial  cholecystectomy  audits  and  the 
proximate  results  i.e.,.  positive  or  negative  change  demonstrated 
which  have  been  discussed  in  th'is  paper  underscore  the  potential 
the  MCE  study  has  for  revealing  removable  problem  areas  relating 
to  medical  care  process  and  outcome. 
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4. 7  Summary  and  Conclusions  for  Chapter  4 

The  preceeding  subchapters  have  described  an  evaluation  of 
the  process  and  the  proximate  results  of  MCE  studies  in  four  topic 
categories.    This  section  of  the  evaluation  was  completed  for  the  pur- 
pose of  obtaining  a  detailed  examination  of  the  effectiveness  of  the 
MCE  as  a  tool  for  improving  the  quality  of  care. 

As  noted  in  the  section  on  methodology,  an  attempt  was  made 
to  review  every  existing  completed  MCE  study  which  had  been  performed 
for  four  topic  areas  (myocardial  infarction,  pneumonia,  appendectomy, 
and  cholecystectomy).    Given  the  time  constraints  of  this  study  and 
the  relatively  small  amount  of  restudy  activity  that  has  been  imple- 
mented by  the  PSROs  and  their  hospitals  to  date,  it  was  possible  to 
locate  only  a  limited  sample  of  studies  which  met  the  selection  cri- 
teria for  evaluation.    Of  65  studies  requested  from  and  submitted  by 
the  PSROs,  53  studies  contained  sufficient  data  to  permit  evaluations 
by  the  impact  rating  system  described  in  this  report.    The  remaining 
12  of  the  65  studies  submitted  were  not  evaluated  because  they  did  not 
include  sufficient  material  on  which  to  base  the  necessary  judgments. 
The  distribution  of  studies  evaluated  is  as  follows: 

Myocardial  Infarction     -  23  studies 

Pneumonia  -  11  studies 

Cholecystectomy  -    7  studies 

Appendectomy  -  12  studies 

In  terms  of  process,  virtually  all  of  the  studies  evaluated 
(35  of  37)  received  acceptable  scores.    However,  as  noted  in  the  previous 


-107- 


sections,  very  liberal  standards  were  applied  in  scoring  on  the  study 
process  criteria.    If  the  study  data  received  from  the  hospitals  are 
representative  of  all  audits  conducted,  clearly,  the  PSROs  and  the 
hospital  could  and  should  be  encouraged  to  be  more  rigorous  in  their 
approach  to  designing,  implementing,  recording  and  interpreting  audit 
results. 

Additionally,  as  was  also  noted  previously,  hospital  audit 
committees  do  not  routinely  document  the  implementation  of  recommended 
educational  or  remedial  actions.    It  is  felt  that  such  documentation  is 
of  great  importance  in  that  it  enables  a  hospital  to  determine,  over  time, 
what  types  of  actions  are  the  most  effective  relative  to  eliminating  pro- 
blem areas  in  medical  care  process  or  patient  outcomes.    For  the  purposes 
of  this  evaluation,  given  the  lack  of  documentation,  it  was  assumed, 
unless  there  was  clear  evidence  to  the  contrary,  that  all  recommended 
actions  had  been  implemented.       All  studies  were  given  acceptable  scores 
for  implementing  actions  related  to  identified  problem  areas  in  medical 
care  process  or  patient  outcomes. 

In  summary,  while  the  studies  evaluated  received  acceptable 
scores  for  process  and  implementation  of  actions,  it  was  necessary  to 
use  relatively  liberal  standards.    The  MCE  process  and  the  documentation 
that  recommended  actions  have  been  taken  should  be  implemented  more  sys- 
tematically and  rigorously. 
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Though  MCEs  were  at  times  lacking  in  terms  of  their  process, 
the  evaluation  clearly  indicates  that  MCEs  can  be  successful  in  reveal- 
ing and  eliminating  problem  areas  relating  to  the  medical  care  process 
and  to  patient  outcomes.    Thus,  while  hospitals  and  PSROs  may  not  always 
be  carrying  out  the  MCE  process  as  rigorously  as  desired,  those  that  are 
done  well  appear  to  have  the  desired  results.    The  MCEs  evaluated  here 
demonstrated  that  they  achieve  two  of  the  primary  goals  of  the  MCE  com- 
ponent of  the  PSRO  Program:  1)  identifying  problem  areas  in  medical  care 
process  and  patient  outcomes  where  they  exist;  2)  removing  or  ameliorating 
these  problem  areas  (reducing  deficiency  rates). 

In  the  following  4  tables  (XIX-XXII)  the  categories  of  deficiencies 
found  for  each  of  the  four  topics  evaluated  are  organized  by  problem  areas 
within  medical  care  process  and  patient  outcomes.    The  six  areas  include: 

•  Failure  to  complete  routine  hospital  care 

•  Failure  to  provide  adequate  diseases  specific 
care:  Diagnostic 

t     Failure  to  provide  adequate  disease  specific 
care:  Treatment 

t     Inappropriate  utilization 

#•    Unjustified  outcomes 

•  Failure  to  document  adequately 

Only  a  few  problem  areas  disclosed  during  these  studies  were 
related  to  delivery  of  routine  hospital  care.    Failure  to  complete  such 
procedures  as  routine  urine  analys'es,  or  chest  x-rays,  or  the  measuring 
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Table  XK 


Categories  of  Deficiencies  Found  in  Ten  MCE  Studies  on  Myocardial  Infarction  Organized 

by  Problem  Area 


PROBLEM  AREA 

EXAMPLE  OF  DEFICIENCIES  WITHIN 
PROBLEM  AREA 

NUMBER  OF  STUDIES 
IN  WHICH  DEFICIENCY 
OCCURRED 

NUMBER  OF  CATEGORIES 

OF  DEFICIENCIES 
WITHIN' PROBLEM  AREA 

Failure  to  complete 
routine  hospital 
care 

Failure  to  obtain  routine  urine  analysis 

1 

1 

Failure  to  provide 
adequate  disease 
specific  care; 
diagnostic 

Failure  to  obtain  EKG  on  admission 
Errors  1n  EKG  Interpretation 
Failure  to  repeat  enzymes  studies  1n 

Instances  of  elevated  enzymes 
No  EKG  during  last  week  documenting 

Improvement 

1 
1 
1 

1 

.     Failure  to  provide 
>     adequate  disease 

specific  care: 

treatment 

Failure  to  admit  patient  to  CCU 

Late  or  no  electronic  monitoring  of  patient 

Failure  to  maintain  I.V.  line 

PVC's  but  no  treatment 

7 
6 
5 
1 

• 

Inappropriate 
uti 1 i  zatlon 

Unjustified  outcomes 

Mortal i  ty 

6 

1 

rai  lure  to  adequately 
document 

no  cnest  x-ray  report  on  criarts 
Failure  to  document  patient  on  monitor 

within  30  minutes 
Failure  to  document  patient  on  I.V. 

within  30  minutes 
Failure  to  document  patient  ambulatory 
Inadequate  documentation  of  discharge 

1 nstructlons 

1 
1 

5 

1 

3 

5 

Table  XX 


Categories  of  Deficiencies  Found  in  Ten  MCE  Studies  on  Penumonia  Organized 

by  Problem  Area 


PROBLEM  AREA 

lami'II  Lt  Ul    UtrlLltNtlES  WlTHIN 
PROBLEM  AREA 

NUMBER  OF  STUDIES 
IN  WHICH  DEFICIENCY 
OCCURRED 

NUMBER  OF  CATEGORIES 

OF  DEFICIENCIES 
WITHIN  PROBLEM  AREA 

Failure  to  complete 
routine  hospital  care 

No  blood  pressure  and  weight  taken 
on  admission 

1 

1 

Failure  to  provide 
adequate  disease 
specific  care: 
di  a  anos  t 1  r 

No  positive  x-ray  diagnosis 

No  positive  x-ray  plus  respiratory 

distress  or  inability  to  cope  out 

of  hospital  as  indicators  for 

admission 
No  positive  culture  for  pathogens 
No  repeated  x-ray  demonstrating 

improvement 

3 
1 

7 

2 

4 

Failure  to  provide 

fl  f|p  Own  f"P    fll  cnncn 

specific  care: 
treatment 

Antibiotic  therapy  1n  absence  of 

bacterial  investigation 
Patient  not  afebrile  for  48  hours  prior 

to  discharge 
No  resolved  or  clearing  x-ray  at 

discharge 

4 
1 
1 

3 

Inappropriate 

II f"  1  1  i  7,1  flnn 

Length  of  stay  outside  of  norms 

• 

1 

1 

Unjustified 
outcomes 

- 

- 

Failure  to  adequately 
document 

No  documentation  of  clinical  Improvement 
No  documentation  of  clinical  Impression 

of  pneumonia 
No  documentation  of  follow-up  discharge 

3 
1 

4 

3 

Table  XXI 

Categories  of  Deficiencies  Found  in  Ten  MCE  Studies  on  Appendectomy 

Organized  by  Problem  Area 


PROGRAM  AREA 

EXAMPLE  OF  DEFICIENCIES  WITHIN 
PROBLEM  AREA 

NUMBER  OF  STUDIES 
IN  WHICH  DEFICIENCY 
OCCURRED 

NUMBER  OF  CATEGORIES 

OF  DEFICIENCIES 
WITHIN  PROBLEM  AREA 

Failure  to  complete 
routine  hospital  care 

Routine  chest  x-ray  not  done 

1 

1 

Failure  to  provide 
adequate  disease 
specific  care: 
diagnostic 

Patient  did  not  meet  criteria  for 
justified  admission 

1 

1 

Failure  to  provide 
adequate  disease 
specific  care: 
treatment 

Patient  not  on  regular  diet  with 

normal  bowel  function  at  discharge 
Patient  did  not  receive  home  care  orders 

1 
1 

2 

Inappropriate 
ut1 1 1  zation 

Length  of  stay  outside  of  norms 

1 

1 

Unjustified 
outcomes 

Surgery  not  justified  by  sufficient 

signs  and  symptoms 
Gangrenous  or  perforated  appendix 

3 
1 

2 

Failure  to  adequately 
document 

Failure  to  document  wound  status  at 
discharge 

Failure  to  document  patient  on  full  diet 
at  discharge 

0 
1 

2 

Table  XX  II 

Categories  of  Deficiencies  Found  in  Seven  MCE  Studies  on  Cholecystectomy 

Organized  by  Problem  Area 


PROBLEM  AREA 

EXAMPLE  OF  DEFICIENCIES 
WITHIN  PROBLEM  AREAS 

NUMBER  OF  STUDIES 
IN  WHICH  DEFFICIENCY 
OCCURRED 

NUMBER  OF  CATEGORIES 
OF  DEFICIENCIES 
WITHIN  PROBLEM  AREA 

Failure  to  complete 
routine  hospital  care 

Failure  to  provide  adequate 
disease  specific  care: 
diagnostic 

No  record  of  pre-operati ve  cholecys- 
togram  confirming  diagnosis  prior  to 
operation 

3 

EKGs  indicated  but  not  done 
Electrolyte  studies  indicated  but 
not  done 

Fasting  blood  sugar  levels  abnormal 
Non-performance  of  operative 
cholongiography 

2 
1 

1 
1 

5 

( 
i 
1 

Failure  to  provide  adequate 
disease  specific  care: 
treatment 

- 

- 

- 

Inappropriate  utilization 

Unjustified  outcomes 

No  satisfactory  wound  healing 

Wound  dehlscience 

No  positive  pathology  report 

2 
1 
1 

3 

Failure  to  adequately 
document 

No  documentation  that  patient  was 

tolerating  diet 
Progress  notes  inadequate  to  justify 

variations 

1 
1 

2 

^  •  o 


of  blood  pressure  and  weight  on  admission  are  examples  of  this  type  of 
deficiency.    Likewise,  only  a  few  problems  were  highlighted  in  the  area 
of  utilization.     Ihjjustified  outcomes  such  as  mortality,  unnecessary 
surgery,  and  wound  dehiscence,  were  problem  areas  occasionally  identified. 
The  main  problem  areas  revealed  by  these  studies,  however,  related  to 
the  failure  to  provide  adequate  disease  specific  care  in  terms  of  diag- 
nosis, the  failure  to  provide  adequate  disease  specific  treatment,  and 
the  failure  to  document  adequately. 

Table  XXIII  summarizes  theMCEs  (of  the  37  evaluated)  in  which 

deficiencies  were  found  and  the  number  of  categories  of  deficiencies  (of 
the  46  categories  found  across  all  studies)  for  each  problem  area.  The 
summarized  data  included  in  Table XXIII  emphasize  the  fact  that  the  sample 
of  studies  evaluated  for  this  study  did,  in  fact,  disclose  problems  in 
medical  care  process  and  in  patient  outcomes. 

The  next  point  of  interest  is  that  theseMCE  studies  were  not 
only  successful  in  highlighting  problem  areas.    They  were  also  successful 
63%  of  the  time  in  eliminating  an  adequate  number  of  all  identified 
deficiencies.    Of  the  37  studies  evaluated,  25  received  acceptable  scores 
for  eliminating  deficiencies,,  that  is,  either  50%  of  unjustified  variation 
was  removed  or  no  more  than  10%  of  medical  practice  or  outcomes  remained 
out  of  compliance  with  a  standard.    The  impact  data  is  somewhat  inconclusi 
in  terms  of  what  types  of  problem  areas  had  the  greatest  number  of  defi- 
ciencies still  remaining  at  the  end  of  the  reaudit.    The  deficiencies 
remaining  seemed  to  fall  primarily  in  the  problem  area  of  failure  to 
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Table  XXIII 


Distribution  of  the  Number  of  Studies  in  Which  Deficiencies  Occurred  By  Problem 
Area  and  the  Number  of  Categories  of  Deficiencies  within  Each  Problem  Area 

For  All  MCE  Studies  Considered 


PR001.EM  AREA 

NUMBER  AND  PERCENT  OF  37  STUDIES  IN  WHICH 
DEFICIENCIES,  OCCURRED 

NUMBER  AND  PERCENT  OF  46  CATEGORIES 
OF  DEFICIENCIES  WITHIN  PROBLEM  A"REA 

Number 

Percent  of  Total  Number 

Number 

Percent  of  Total  Number 

Failure  to  complete 
routine  liospl  tal 
care 

mi  - 1  Total 
P-1  3 
A-l  J 
C-0 

MI-2.7  Total 
P-2.7              0  , 
A-2.7  ■■' 
C-0 

MI-1  Total 

M  3 
A-1  J 

C-0 

MI-2.2    •  Total 

A-2.2  6,6 
C-0 

la  1  lure  to  provide 
adequate  disease 
spec! f 1c  care:  . 

•  UIu4JIIUjI.IL, 

MI-4 
MS  26 
A-l 
C-0 

MI-10.0 
P-35.0  70.1 
A-2.7 
C-21.6 

MI-4 
C-5 

MI-0. 7 

P-0-7  30.4 

A-2.2 

C-11.0 

la  11 ure  to  provide 
adequate  disease 
spec! f Ic  care: 
treatment 

MI-  19 
P-6 

A- 2  27 
C-0 

MI-61.3 
P-16.2 

A-  5.4  72.9 
C-  0 

MI-4 
P-3  g 
A-2  * 
C-0 

Ml-0.7 

A-4  .3 
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♦Reproduced  from  "Analysis  of  the  Results  (Process  and  Immediate  Impart)  of  MCEs  Done 
in  Four  Major  Topfo  Areas,"  August  22,  1'  i 


adequately  document,  and  the  problem  area  of  failure  to  provide  ade- 
quate disease  specific  diagnoses.    However,  a  point  which  may  be  parti- 
cularly important  and  encouraging  is  that  all  outcome  deficiencies  were 
eliminated.    For  this  sample  of  studies,  despite  their  varying  effective- 
ness in  eliminating  medical  care  process  deficiencies,  the  MCEs  examined 
succeeded  in  removing  all  deficiencies  in  the  most  critical  category, 
patient  care  outcomes. 

The  results  of  the  studies  evaluated  lead  to  the  conclusion  that 
when  audits  are  reasonably  well  done,  more  than  half  of  any  deficiency 
identified  can  be  eliminated  or  reduced  to  a  point  where  variation  is  no 

more  than  10%.     If  BQA  can  encourage  "a  higher  level  of  reaudit  activity  and 
more  rigorous  implementation  of  the  entire  MCE  process,  the  potential  of 
the  MCE  study  for  disclosing  and  eliminating  problem  areas  can  be  further 
demonstrated.    Also,  with  more  MCEs  to  examine  the  actual  impact  of  the 
types  of  results  pointed  out  here  can  be  measured  with  greater  specificity 
and  accuracy. 
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5.0 


AN  EVALUATION  OF  THE  POTENTIAL  IMPACT  OF  THE  OVERALL 
MCE  PROGRAM 


The  concepts  underlying  the  medical  audit  are  in 
the  finest  tradition  of  medicine  and  go  back  to  the 
very  beginning  of  the  profession.    Medicine  is  a  true, 
learned  profession.    The  physician  is  unselfishly 
dedicated  to  the  welfare  of  his  fellow  men.    Now  that 
the  term  "professional"  is  applied  loosely  to  almost 
anyone  who  is  paid  for  whatever  he  does,  it  is  impor- 
tant that  the  attributes  of  a  learned  profession  be 
clearly  defined.    High  on  the  list  is  the  willingness 
to  engage  in  self-devaluation,  self-criticism,  and 
self-discipline. 

—  C.  W.  Eisele,  M.D. 


5.1  Introduction 

The  Medical  Care  Evaluation  Program  as  currently  implemented 
is  a  means  for  physicians  to  review  the  technical  quality  of  professional 
care  provided  by  their  peers  and  other  providers  of  services  in  their 
hospitals.    This  is  accomplished  by  comparing  information  from  patient 
records  with  standards  of  care  which  physicians  in  individual  hospitals 
have  themselves  established.    It  is  the  aim  of  the  MCE  process  to  detect 
and  remove  problem  areas  in  the  medical  care  process  and  to  then  confirm 
that  improvement  in  the  quality  of  care  with  a  reaudit. 

The  purpose  of  this  volume  is  to  evaluate  the  MCE  Program  as 
it  identifies  and  eliminates  problem  areas.    There  are  several  ways  this 
can  be  done.    First,  one  may  attempt  to  measure  the  effect  of  MCEs  on 
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the  health  status  of  a  population.    Given  the  difficulty,  however,  of 
measuring  community  health  status  with  any  specificity,  the  many  non- 
medical factors  which  influence  that  status,  and  the  early  stage  of 
the  PSRO  Program,  this  approach  is  not  feasible. 

Alternatively,  one  may  look  for  improvement  in  the  health 
status  of  those  members  of  the  community  who  have  been  treated  in  the 
hospital.    Few  studies  thus  far  have  examined  the  long-term  outcomes 
of  hospital  treatment,  and  it  is  likely  that  once  the  patient  leaves 

the  hospital  the  difficulties  of  follow-up  will  prevent  such  measures 

2 

from  becoming  a  routine  part  of  the  medical  audit  for  some  time. 

A  third  approach  to  MCE  assessment  is  to  examine  changes  in  health 
system  performance  which  have  come  about  as  a  result  of  medical  audits,  The 
anecdotes  presented  in  Cahpter  4  demonstrate  that  individual  MCEs  can  bring 
about  dramatic  improvements  in  the  process  of  medical  care  and  proximate 
patient  outcomes  on  a  case  by  case  basis.    However,  we  do  not  yet  have  suf- 
ficient data  to  evaluate  the  impact  of  the  whole  MCE  Program.    This  Chapter 
estimates  that  overall  impact  by  examining  how  well  PSROs  are  selecting 
MCE  topics  and  how  thoroughly  they  are  instituting  MCE  follow-up  procedures, 

The  rationale  of  this  chapter  is  as  follows:    The  MCE  Program 
can  have  impact  on  overall  medical  care  or  patient  outcomes  only  if 
present  MCE  activity  has  potential  to  identify  and  remove  important 
problem  areas. 

Without  adequate  topic  selection,  MCEs  have  little  chance  of 
finding  problem  areas;  without  adequate  follow-up  there  is  no  assurance 
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that  the  problem  areas  found  will  be  acted  upon  or  that  actions  taken 
are  appropriate.    Thus,  the  MCE  Program's  likelihood  of  having  impact 
at  present  is  only  as  good  as  the  methods  used  by  PSROs  to  date  in 
these  two  areas.    Topic  selection  is  the  subject  of  the  first  part  of 
this  chapter;  follow-up  is  discussed  in  the  second  part. 

5-2  A  Conceptual  Framework  for  Evaluating  PSRO  Practices  In 

MCE  Topic  Selection 

Any  attempt  to  assure  the  quality  of  American  medical  care, 
even  if  its  purpose  is  limited  to  treatment  provided  in  short-term  acute- 
care  hospitals,  is  necessarily  a  large  and  complex  undertaking.    In  a 
recent  year,  for  example,  the  National'  Center  for  Health  Statistics 
estimates  that  22  million  Americans  -  over  10%  of  the  population  -  were 
hospitalized  in  short-stay  hospitals  for  a  total  of  235  million  patient- 
days.3    Care  was  provided  to  these  patients  through  a  complex  technological 
and  organizational  structure  with  a  highly  sophisticated  division  of  labor. 
While  it  is  the  goal  of  the  PSRO  Program  as  a  whole  to  promote  "the  effec- 
tive, efficient,  and  economical  delivery  of  health  care  services  of  proper 
quality"  for  those  patients  whose  care  is  federally  funded,  physicians 
carrying  out  MCE  reviews  must  consider  the  quality  of  care  provided 
to  all  patients,  regardless  of  the  source  of  payment. 

The  MCE  Program  can  be  conceived  of  as  an  organized  means  of 
monitoring  the  countless  operations  of  a  very  large  health  care  system, 
comparing  performance  with  accepted  standards,  feeding  back  to  the 
system  recommendations  for  improvement  of  quality  of  care,  and  assuring 
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that  these  recommendations  are  acted  upon.    There  is  no  doubt  that 
the  vast  majority  of  the  elements  of  care  are  being  carried  out 
effectively:  the  premise  of  the  MCE  Program  is  that  unknown  -  perhaps 
randomly  distributed  -  problem  areas  remain  and  that  systematic  com- 
parison of  the  elements  of  care  in  selected  topic  areas  with  explicit 
standards  can  detect  them  and  lead  to  their  elimination.    It  is  the  pur- 
pose of  this  chapter  to  develop  a  methodolgy  for  rating  MCE  topics  accord 
ing  to  the  probability  that  they  will  reveal  problems  in  the  medical 
care  delivery  system  under  review.    The  methodology  will  then  be  used 
to  evaluate  the  adequacy  of  topic  selection  among  the  studies  in  the 
sample  under  consideration  which  consists  of  all  MCEs  in  the  twenty 
most  frequent  topic  categories.    In  the  discucssion  which  follows,  all 
clinical  judgments,  particularly  factor  weights,  have  been  made  by 
a  panel  of  three  physicians:  the  Project  Officer,  the  Project  Monitor, 
and  the  MCE  Project  Consultant. 

An  indicator  of  the  probable  impact  of  an  MCE  done  in  a  particu 
lar  topic  category  has  been  developed.    The  indicator  is  "computed"  for 
each  topic  category  by  ranking  it  according  to  three  criteria: 

1 .    Chance  of  Finding  and  Removing  Randomly  or  Sys- 
tematically Distributed  Problem  Areas 

The  topic-  category  whose  scope  includes  the  most 
patients,  physicians,  and  costly  services  will  most 
likely  contain  MCEs  that  elucidate  important  pro- 
blem areas.    A  further  constraint  is  the  MCEs  in 
these  categories  will  have  impact  only  if  the  pro- 
blem areas  can  be  realistically  removed. 
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2.  Situation  Specific  Rationale 

The  topic  most  commonly  selected  in  response  to 
local  needs  (based  on  known  hospital  difficult- 
ies) will  most  likely  target  the  audit  to 
existing  problem  areas . 

3.  Individual  MCE  Topic  Specificity 

The  MCE  topics  which  are  selected  with  objectives 
that  focus  "the  scope  of  the  topic  will  be  most 
likely  to  be  successful  in  sorting  out  deficiencies 
into  discrete  problem  areas. 

An  individual  MCE  Topic  may  be  judged  valid  (and  be  expected 
to  point  out  problem  areas)  if  the  scope  of  the  Topic  Category  is  large 
enough  to  ensure  the  discovery  of  random  or  systematically  distributed 
remediable  problem  areas.    The  value. of  the  indicator  increases  and 
the  individual  topic  is  more  validly  chosen  if  the  topic  is  specifically 
focused,  further  permitting  the  disclosure  of  problem  areas.  Lastly, 
the  topic  may  be  validly  chosen  if  the  situation  specific  factors  war- 
rant an  MCE,  even  if  the  scope  of  the  topic  category  is  small. 

The  meaning  of  these  criteria,  their  operational  definition, 
and  their  application  will  be  presented  in  the  next  three  sections. 
The  manner  in  which  they  are  combined  into  an  overall  topic/category 
rating  system  will  be  discussed.    The  20  most  frequent  topic  categories 
will  be  ranked  according  to  their  likelihood  of  containing  potentially 
high  impact  MCE  (MCEs  which  identify  important  problem  areas).  The 
pattern  of  overall  PSRO  topic  selection  process  will  then  be  assessed 
by  use  of  the  topic  category  rating  system. 
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In  terms  of  the  conceptual  system  to  be  described,  an 
individual  MCE  study  topic  is  determined  to  be  valid  if  it  falls 
into  either  outcome  three  or  seven  on  the  following  algorithm  or 
decision  tree  chart. 

Chart  IV 


Chance  of  Finding 
Problem  Areas 


Criterion  3: 
Individual  Topic  Specificity 


Criterion  2: 
Situation  Specific  Factors 


Criterion  3: 
Individual  Topic  Specificity 


5.3  Criterion  1:    Chance  of  Finding  and  Removing  Randomly 

or  Systematically  Distributed  Problem  Areas 

A  well-chosen  MCE  topic  category*  should  have  a  high  probability 
of  finding  existing  problems  in  medical  care  process  or  patient  outcomes. 
The  topic  category  is  most  likely  to  facilitate  problem  identification 
if  it  maximizes  two  attributes:  a)  if  the  category  is. of  sufficient 
scope  that  it  involves  a  large  proportion  of  the  health  delivery  system, 
as  measured  by  the  numbers  of  patients,  physicians,  and  dollars  involved 
annually;  and  b)  if  there  is  agreement  on  indications  for  treatment  and 
modes  of  treatment,  and  treatment  is  likely  to  lead  to  improvement  in 
problem  areas  such  as  the  patient's  health'status.    MCEs  in  topic  cate- 
gories with  great  importance  and  magnitude  can  be  expected  to  reveal 
problem  areas,  if  they  exist,  which  are  of  proportionate  importance. 
However,  MCEs  even  in  topic  categories  of  large  scope  can  have  little 
impact  if  there  is  not  general  consensus  concerning  application  of  pro- 
cedures for  diagnosis  and  treatment  or  if  treatment  is  ineffective  in 
removing  health  problems  of  the  patient. 

The  chance  of  a  topic  category  to  contain  MCEs  that  find 

randomly  or  systematically  distributed  problems,  then,  can  be  rated 

by  measuring  and  combining  four  factors: 

Factor  I:        Size  of  the  patient  population  directly 
and  potentially  affected  by  the  topic 
category.. 

*  As  explained  in  Chapter  III,  a  topic  category  is  an  aggregation 
of  individual  MCE  topics,  generally  by  diagnostic  or  procedural 
category. 
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Factor  II 


Factor  III: 


Factor  IV: 


Size  of  the  provider  population  directly 
and  potentially  affected  by  the  topic 
category. 

The  direct  economic  cost  of  the  disease 
entity  or  hospital  procedures  audited 
within  the  topic  category. 

Efficacy  as  reflected  by  degree  of  agreement 
on  indications  for  treatment  and  modes  of 
treatment,  and  the  likelihood  that  treatment 
will  lead  to  health  benefit. 


Factor  I  -  Size  of  Patient  Population 

Factor  I,  the  size  of  the  patient  population  directly  or 
indirectly  affected  by  a  topic  related  category  consists  of  two  com- 
ponents.   Each  component  is  assigned  a  weighted  component  value  reflec- 
ting its  relative  importance,  and  component  values  are  combined  to  yield 


a  factor  value. 


Components 

The  factor  value  of  Factor  I,  size  of  the  patient  population, 


is  computed  on  the  basis  of  two  components.    These  are: 

a)  Annual  hospital  care  episodes,  as  indicated  by  the 
total  number  of  discharges  from  short  stay  hospitals, 
in  1975,  by  primary  diagnosis.    Data  used  here  were 
compiled  by  the  Hospital  Statistics  Branch  of  the 
National  Center  for  Health  Statistics,  and  have  not 
yet  been  published.    Data  on  the  frequency  of  emer-_ 
gency  room  discharges  were  obtained  from  1975  hosptial 
statistics  reported  by  the  American  Hospital  Associa- 
tion. 

b)  Prevalence  of  the  medical  condition.    Prevalenece  data 
were  compiled  by  the  Health  Interview  Survey  of  the  . 
National  Center  for  Health  Statistics,  by  the  Biometry 
Branch  of  the  National  Heart  and  Lung  Institute  and 
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and  by  the  Arthritis  and  Digestive  Diseases 
Branches  of  the  National  Institute  on  Arthritis, 
Metabolism,  and  Digestive  Diseases. 

Factor  Value  Computation 

The  total  number  of  hospital  dischrages  in  each  topic  category 
indicates  the  annual  number  of  hospital  care  episodes  related  to  the 
disease  entity  or  medical  procedure  being  considered.    The  size  of  the 
patient  population  potentially  affected  is  indicated  by  the  prevalence 
of  the  medical  condition.    Where  both  prevalence  and  hospital  care 
episode  data  provide  relevant  information,  these  components  are  assigned 
component  values  which  are  combined  to  arrive  at  the  factor  value  for 
each  topic  category. 

For  certain  topic  categories,  such  as  hysterectomy  and  delivery, 
prevalence  data  do  not  provide  meaningful  information.    In  these  instances 
the  component  value  for  hospital  discharge  rate  is  given  additional  weight 
to  equal  the  potential  combined  weight  allowed  for  both  components,  and 
provides  the  factor  value. 

To  provide  a  basic  rank  value,  topic  categories  are  arranged 
in  inverse  rank  order  within  each  component  rating.    Thus  the  topic 
category  of  greatest  magnitude  receives  the  highest  rank  value.  In 
the  case  of  the  sample  considered  by  this  report,  the  highest  rank  value 
is  20  since  there  are  20  topic  categories. 

Component  values  are  computed  by  multiplying  a  weight  of  .4 
by  the  topic  category's  rank  for  the  hospital  care  episode  component, 
and  a  weight  of  .3  times  the  rank  for  the  prevalence  component.  This 
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yields  a  combined  weight  of  .7  for  the  two  components  of  this  factor. 
The  two  component  values  are  then  added  to  yield  a  score  on  Factor  I 
for  each  topic  category  under  consideration.    The  weighting  assigned  to 
the  hospital  care  episodes  component  is  somewhat~higher  than  that  assigned 
to  the  prevalence  component,  as  the  annual  number  of  hospital  care  episodes 
was  judged  by  the  physician  panel  to  be  the  more  important  indicator  of 
the  magnitude  of  the  patient  population  affected.    When  there  are  no 
prevalence  rankings,  hospital  care  episodes  rankings  are  multiplied  by 
the  weights  of  .7.    Since  there  are  only  seven  prevalence  indicators, 
their  rank  values  were  computed  on  as  basis  of  1  to  7  and  these  7  values 
were  evenly  distributed  over  the  range  of  20  rankings  with  an  interval 
of  2.9  between  each. 

As  displayed  on  Table  XXIV  when  factor  values  were  computed  for 
the  20  most  frequently  audited  topic  categories,  emergency  room  was  found 
to  affect  the  largest  patient  population.    Delivery,  tonsillectomy  and 
adenoidectomy,  hysterectomy,  pneumonia,  breast  lesions,  appendectomy, 
cholecystectomy,  ulcer  and  inguinal  hernia  were  also  found  to  affect  large 
patient  populations. 

Factor  II:  Size  of  Provider  Population  Directly  and 
Indirectly  Affected  by  A  Topic  Category 

Factor  II,  the  size  of  the  provider  population  directly  and 

indirectly  affected  by  the  topic  categories  under  consideration,  is 

judged  equal  in  importance  to  Factor  I  in  determining  the  relative 
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Table  XXIV 
FACTOR  I:    Size  of  Patient  Population 
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potential  impact  of  topic  categories  and  has  been  given  equal  weight. 
Components 

Factor  II  is  comprised  of  two  components.    These  are: 

a)  The  overall  number  of  physicians  who  are  potentially 
involved  in  health  care  delivery  for  the  indicated 
disease  entity  or  medical  procedure. 

b)  The  number  of  physicians  and  special  facilities  or 
services  personnel,  such  as  coronary  care  unit, 
operating  room  or  other  ancillary  personnel  involved 
in  each  episode  of  care  for  the  topic  category. 

The  first  component,  the  number  of  physicians  potentially 
involved  in  delivering  care,  indicates  the  total  number  of  physician  pro- 
viders in  the  nation  who  logically  might  be  expected  to  be  involved  in 
the  delivery  of  care  for  this  topic  category. 

The  second  component,  the  number  of  physicians  and  special 
facilities  or  services  personnel  involved  in  each  episode  of  care, 
indicates  the  size  of  the  provider  team  potentially  required  for  delivery 
of  care  for  each  health  care  episode  within  a  topic  category.  Provider 
team  here  refers  to  physicians  and  special  facilities  and  services  per- 
sonnel.   It  does  not  include  nurses  who  do  not  work  in  special  facilities 
or  services. 

Factor  Value  Computation 

Factor  II  is  considered  to  be  of  equal  importance  with 
Factor  I.  To  assure  this  the  same  factor  weight  of  0.7 
was  assigned.  This  resulted  in  a  maximum  factor  value 
range  for  provider  population  equal  to  one  third  of  the 
magnitude  of  problem  rating.  This  was  achieved  by  allowing 
factor  of  0.7  to  14. 
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Of  the  two  components  of  Factor  II,  the  first  com- 
ponent, total  potential  physician  involvement  per 
topic  category,  was  considered  to  be  more  important 
and  is  weighted  more  heavily  than  the  second.  Com-  j 
ponent  1  was  assigned  a  weight  of  0.4;  component  2  ! 
was  assigned  a  weight  of  0.3. 

No  reliable  data  are  available  for  either  total'  numbers  of 
providers  involved  or  the  providers  per  episode  for  each  of  the  topic 
categories.    Therefore,  a  matrix  was  constructed  of  physician  providers 
by  specialty  who  would  normally  be  expected  to  be  involved  in  some  aspect 
of  delivery  of  care  for  each  of  the  topic  categories.    This  matrix  is 
shown  in  Table  II.    Provider  specialty  or  service  areas  affected  by 
each  of  the  topic  categories  were  determined  by  the  Project  Office  in 
collaboration  with  the  Project  Monitor,  both  of  whom  are  physicians. 
Figures  for  the  number  of  physicians  in  each  of  the  specialty  categories 
were  obtained  from  the  AMA's  "Physician  Distribution  and  Medical  Licen- 
sure, 1975." 

Rank  assignments  for  component  1  are  determined  by  summing 
the  total  number  of  providers  in  each  specialty  or  service  area  potentially 
affected  by  the  topic' category  under  consideration.    For  the  twenty  most 
frequently  audited  topic  categories,  these  determinations  were  made  on 
the  basis  of  the  information  presented  in  Table  XXV. 

To  obtain  a  comparative  index  of  provider  involvement  per 
episode  of  care  for  each  of  the  topic  categories,  some  adjustment  was 
required  beyond  merely  totalling  the  affected  specialties  and  service 
areas  shown  on  Table  XXV.    For  each  topic  category  a  primary  care  physician 
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Table  XXV 

Probable  Provider  Involvement  in  MCE  Study  Topic  Areas 


TOPIC  RELATED  CATEGORY 

GP  (51270) 

Immun.  (1517) 

CV  (5584) 

Gastro  (1814) 

Int.  Med.  (43936) 

Ped  (18546) 

<rT 

CO 
LO 

P 

Ai 

Surg  (28460) 

OB/GYN  (19742) 

Opth  (10320) 

Ortb  (10207) 

OTO  (5132) 

Plast  (2078) 

Th.S.  (1729) 

CI 

w 

a 
<i 

& 
o 

CO 

p 

n. 

f— 1 

o 
f-1 

Psych  (18963) 

PMR/OM  (2824) 

Rad  (12828) 

Emer.  Rm.  (15000) 

Other  Spec.  (15791) 

Ancillary  Services 

Producer  Participation 
Per  Episode 

Myocardial  Infarction 

• 

0 

X 

0 

• 

0 

x 

0 

0 

0 

0 

0 

0 

X 

0 

0 

0 

0 

0 

X 

X 

0 

0 

• 

5.5 

Pnuemonia 

• 

X 

u 

u 

• 

X 

X 

u 

U 

0 

ft 

u 

ft 

ft 

u 

ft 

0 

0 

0 

ft 

u 

ft 

u 

X 

X 

n 
u 

0 

U 

Cholecystectomy 

X 

0 

0 

X 

0 

0 

0 

• 

0 

0 

0 

0 

0 

0 

X 

0 

0 

0 

0 

X 

X 

0 

• 

0 

4.5 

Appendectomy 

X 

0 

0 

X 

0 

0 

0 

• 

X 

0 

0 

0 

0 

0 

X 

0 

0 

0 

0 

0 

X 

0 

« 

0 

4.5 

Hysterectomy 

X 

0 

0 

0 

0 

0 

0 

X 

• 

0 

0 

0 

0 

0 

X 

0 

0 

0 

0 

0 

0 

0 

• 

0 

3.5 

C-Section 

X 

0 

0 

0 

0 

0 

0 

X 

• 

0 

0 

0 

0 

0 

X 

0 

0 

0 

0 

0 

0 

0 

• 

0 

3.5 

Tonsillectomy  &  Adenoidectomy 

X 

0 

0 

0 

0 

• 

0 

X 

0 

0 

0 

• 

0 

0 

X 

0 

0 

0 

0 

0 

0 

0 

• 

0 

4.5 

Hip  Fracture 

0 

0 

0 

0 

0 

0 

0 

X 

0 

0 

• 

0 

0 

0 

X 

0 

0 

0 

X 

X 

X 

0 

• 

• 

5.5 

Diabetes 

X 

0 

X 

0 

• 

X 

0 

X 

X 

X 

0 

0 

0 

0 

0 

0 

X 

0 

0 

0 

X 

0 

0 

• 

6 

Prostatectomy 

0 

0 

0 

0 

0 

0 

0 

X 

0 

0 

0 

0 

0 

0 

x 

• 

0 

0 

0 

0 

X 

0 

• 

• 

4.5 

Breast  Lesions 

X 

0 

0 

0 

0 

0 

0 

• 

X 

0 

0 

0 

X 

0 

X 

0 

0 

0 

X 

X 

0 

0 

• 

0 

5 

Emergency  Room 

0 

0 

0 

0 

X 

X 

0 

X 

0 

0 

X 

0 

X 

X 

0 

0 

0 

0 

0 

0 

• 

0 

0 

4 

Inguinal  Hernia 

X 

0 

0 

0 

0 

X 

0 

• 

0 

0 

0 

0 

0 

0 

X 

X 

0 

0 

0 

0 

0 

0 

• 

0 

4 

Gastroenteritis 

• 

X 

X 

0 

X 

X 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

X 

X 

0 

0 

0 

4 

Ulcer 

X 

0 

0 

,x 

• 

X 

0 

X 

0 

0 

0 

0 

0 

0 

0 

0 

0 

X 

0 

X 

X 

0 

0 

0 

4.5 

Cataract 

X 

0 

0 

0 

0 

0 

0 

0 

0 

• 

0 

0 

0 

0 

X 

0 

0 

0 

0 

0 

0 

0 

• 

0 

3 

Hypertension 

• 

0 

X 

0 

• 

X 

X 

0 

X 

X 

0 

0 

0 

0 

0 

0 

X 

0 

0 

X 

X 

0 

0 

• 

7 

Delivery 

X 

0 

0 

0 

0 

0 

0 

0 

• 

0 

0 

0 

0 

0 

X 

0 

0 

0 

0 

0 

X 

0 

• 

0 

3.5 

Cerebrovascular  Accident 

X 

0 

X 

0 

• 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

X 

X 

X 

X 

X 

0 

0 

• 

5.5 

Congestive  Heart  Failure 

X 

0 

X 

0 

• 

X 

X 

0 

0 

0 

0 

0 

0 

X 

0 

0 

0 

0 

0 

X 

X 

0 

0 

• 

5.5 

was  designated  as  indicated  by  the  darkened  circle  on  the  matrix. 
This  was  given  a  value  of  one.    For  certain  topic  categories  primary 
care  might  be  referred  or  shared  by  different  types  of  physicians. 
Since  other  physician  categories,  designated  as  potentially  involved, 
represent  alternative  providers  in  many  cases,  they  are  accorded  a 
value  of  1/2.    Additional  recognition  is  given  to  those  procedures 
which  require  more  than  normal  non-physician  provider  input,  such  as 
operating  room  personnel,  coronary  care  unit  personnel,  and  providers 
of  extensive  follow-up  services  (therapists,  dieticians,  etc.).  Two 
additional  categories  of  personnel  are  added  for  these  provider  groups 
(special  facilities  and  ancillary  services).    Each  was  counted  as  a 
value  of  one  where  this  addition  is  appropriate. 

As  Table XXVI indicates,  of  the  20  most  frequently  audited 
topic  related  categories,  diabetes  was  determined  to  affect  the  greatest 
number  of  providers.    Hypertension,  cerebrovascular  accident,  congestive 
heart  failure  and  ulcer  also  affect  large  numbers  of  providers. 

Factor  III:  Direct  Economic  Cost 

The  third  factor  considered  in  assessing  the  validity  of  topic 
categories  is  economic  cost.    Evaluating  this  factor  contributes  to 
understanding  the  relative  cost  benefit  of  assuring  the  quality  of 
medical  care  for  various  procedures  and  conditions.    Because  Factor  III 
is  considered  to  be  as  important  as  Factors  I  and  II  in  assessing  a  topic 
category's  potential  impact  on  the  health  status  of  the  population,  it  is 
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Table  XXVI 

Weighted  Comparative  Value  for  Provider  Population  (Factor  II) 


TOPIC  RELATED  CATEGORY 

Number  of 
Physicians 
Affected  (in 
thousands) 

11 IV  t- 1  DC 

Rank 

vllll'l 

Component 
Value 

Providers 
per  Episode 

Inverse 
ivanK 
Order 

Component 
Value 

Factor 
Value 

Myocardial  Infarction  , 

132 

13 

5.  2 

D  •  D 

J.  Urn  o 

o 

in  9 

III.  .' 

Pneumonia 

145 

14.  5 

5. 8 

4  ^ 

1  9 
u  •  u 

<i  n 

Cholecystectomy 

121 

10 

4 

2i  n 
<i .  u 

in  ^ 

9  9 

7  9 

Appendectomy 

128 

11  ^ 

X  X  a  ij 

X  •  I) 

1  n  r; 
1 U.  O 

Q  O 
.5  .  6 

7  Q 
(  .  O 

Hysterectomy 

111 

C\  R 
y>m  o 

9  (X 

9  n 
J.  0 

o 
,i 

u.  y 

O  r 

3.  5 

C-Section 

111 

R  ^ 
u .  o 

9  fi 

9  c 

o 

a 

0.  y 

»».  5 

Tonsillectomy  &  Adenoidectomy 

1  1 

1  Id 

Q 
O 

4.  5 

10.  5 

3.  2 

6.  4 

Hip  Fracture 

81 

4 

1.  G 

5.5 

16.5 

5 

C.6 

Diabetes 

19G 

20 

8 

6 

19 

5.  7 

13.  7 

Prostatectomy 

61 

2 

.8 

4.5 

10.  5 

3.2 

4.0 

Breast  Lesions 

128 

11.5 

4.6 

5 

14 

4,2 

fl.8 

Emergency  Room 

15 

1 

.4 

4 

6 

1.8 

2.2 

Inguinal  Hernia 

116 

9 

3.6 

4 

6 

1.8 

5.4 

Gastroenteritis 

145 

14.5 

5.8 

4 

6 

1.8 

7.6 

Ulcer 

191 

19 

7.6 

4.5 

10.5 

3.2 

10.8 

Cataract 

73 

3 

1.2 

3 

1.0 

0.3 

I.  5 

Hypertension 

182 

18 

7.2 

7 

20 

6 

13.  2 

Delivery 

98 

5 

2 

3.5 

3 

0.9 

2,9 

Cerebrovascular  Accident 

153 

17 

6.8 

5.5 

16.  5 

5 

11.8 

Congestive  Heart  Failure 

150 

16 

6.4 

5.5 

16.5 

5 

11.4 

weighted  equally  with  them  and  contributes  up  to  one  third  of  the 
magnitude  of  the  problem  rating. 

Components 

The  economic  cost  factor  value  is  of  a  composite  of  four 
cost  related  components.    These  are: 

a.      Average  total  charge  per  discharge  by  primary  diagnosis  of 
Medicare  beneficiaries  in  short  stay  hospitals,  exclusive  of  physicians' 
fees,  in  1973.    Though  these  costs  represent  charges  made  to  Medicare 
beneficiaries,  the  relative  magnitudes  of  hospital  charges  by  topic 
categories  are  assumed  to  reflect  relative  magnitudes  of  charges  made 
to  other  types  of  hospital  patients. 

Average  charge  per  discharge  data  were  generated  from  the 
Social  Security  Administration's  "Med  Par"  analysis  of  a  20%  random 
sample  of  billing  forms  for  Medicare  beneficiaries  in  short  stay  hospitals 
in  1973. 4    Costs  displayed  include  all  hospital  service  charges  including 
length  of  stay  and  ancillary  services  charges.    Physicians'  charges  are  no 
included.    Costs  cited  for  topic  categories  in  Classification  A,  Medical, 
reflect  mean  charges  per  discharge  of  non-operated  patients.    Costs  cited 
for  topic  categories  in  Classification  B,  Surgical,  reflect  mean  charges 
per  discharge  of  patients  who  underwent  surgery. 

Data  were  not  available  from  this  source  for  hospital  costs  of 
cesarean  section,  tonsillectomy  and  adenoidectomy ,  delivery,  and  emer- 
gency room.    Costs  for  the  first  three  of  these  procedures  are  not 
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reimbursed  by  Medicare.    The  cost  estimates  cited  for  delivery  and 
tonsillectomy  and  adenoidectomy  were  interpolated  from  average  hospital 
charges  per  day  of  care  for  surgical  procedures,  as  reported  by  the 
Social  Security  Administration,  and  average  length  of  hospital  stay, 
as  reported  by  the  Commission  on  Professional  Hospital  Activities. 
Interpolations  from  this  formula  have  a  standard  error  of  78%  of  one 
standard  deviation.    These  two  estimates  may  therefore  be  in  error  by 
S231  and  $113  respectively.    The  cost  estimate  provided  for  cesarean 
section  was  interpolated  from  hospital  charges  and  physicians  charges, 
as  length  of  stay  data  are  not  available.    This  estimate  may  be  in  error 
by  $549.    Emergency  room  charges,  which  are  not  available  through  the 
Social  Security  Administration  or  the  Insurance  Commission  of  the  State 
of  Maryland,  were  taken  from  a  previous  study  on  the  cost  of  emergency 
room  procedures  in  Cleveland. 

b.    The  50th  percentile  cost  of  physicians'  services  for  hospitalized 
patients  in  the  State  of  Maryland  by  primary  diagnosis  or  procedure,  in 
1973.    Median  physicians'  charges  in  the  State  of  Maryland  were  taken  as 
representative  of  the  relative  magnitude'of  physicians'  charges  per  topic 
category.    Though  physicians'  charges  may  vary  in  different  areas  of  the 
country,  it  is  assumed  that  their  relative  magnitude  will  not,  nor -will 

i 

their  rank  order.    In  other  words,  it  is  assumed  that  in  general  the  most 
expensive  procedures  in  a  given  area  will  also  be  the  most  expensive 
procedures  in  any  other  area.    The  Insurance  Commission  of  the  State 
of  Maryland  maintains  comprehensive  data  on  these  costs  within  that 
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state;  these  data  are  not  available  on  a  national  basis. 

c.  A  none/some  ranking  of  the  chronicity,  or  likelihood  of 
of  repeated  episddes  of  hospital  care,  for  patients  with  conditions 
under  consideration.    The  long-term  direct  economic  cost  of  a  topic 
category  is  influenced  by  that  topic's  chronicity.  To  assure  appropriateness 

and  reliability,  ranking  determinations  were  made  by  the  Project  Officer  in 
collaboration  with  the  Project  Monitor,  both  of  whom  are  physicians. 

d.  The  final  component  considered  in  the  economic  cost  factor  is 
the  likelihood  that  substantial  follow-up  treatment  and  services  can 

be  expected  to  be  associated  with  specified  conditions.    Magnitude  of 
follow-up  costs  are  indicated  by  a  none/some  ranking  which  was  again 
made  b,y  the  Project  Officer  in  collaboration  with  the  Project  Monitor, 
both  of  whom  are  physicians. 

Factor  Value  Computation 

Factor  III  is  considered  to  be  as  important  as  Factor  I  and 
II  from  the  perspective  of  quality  assurance  and  is  given  the  same  weight. 

A  basic  rank  ordering  of  topic  categories  is  determined  by 
an  adding  of  components  a  and  b  which  indicates  the  total  dollar  costs 
per  episode  of  care  within  each  topic  category.    Inverse  rank  order 
values  are  then  multiplied  by  .55,  a  weighting  coefficient  which  is 
employed  to  maintain  the  equality  in  magnitude  between  Factor *I I I  and 
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the  other  two  factors.    The  resulting  component  value  for  total 
dollar  cost  is  then  modified  by  the  addition  of  the  modifier  value 
(2)  for  chronicity  and  the  value  (1)  for  follow-up.  The  combined  com- 
ponent values  of  component  c  and  d  may  represent  up  to  21.4%  of  the 
factor  value  for  the  most  expensive  topic  related  category,  and  up  to 
84.5%  of  the  factor  value  of  the  least  expensive  topic  related  category. 
This  is  felt  to  reflect  the  relative  importance  of  chronicity  and 
follow-up  to  the  direct  economic  cost  of  an  illness.    They  increase 
in  direct  proportion  to  the  size  of  the  sample  being  considered.  Com- 
bining total  dollar  cost  component  values  with  chronicity  and  follow-up 
values  yields  the  factor  value  for  Factor  III. 

Table  XXVEE  demonstrates  that  of  the  20  most  frequently  audited 
topic. categories,  cesarean  section  was  found  to  incur  the  highest  direct 
economic  cost.    Myocardial  infarction,  cholecystectomy,  breast  lesions, 
prostatectomy,  appendectomy,  cerebrovascular  accident,  congestive 
heart  failure,  hysterectomy,  and  cataract  were  also  found  to  incur  high 
direct  economic  costs. 

Factor  IV:  Efficacy 

Efficacy  refers  to  the  potential  for  achieving  positive  or 
acceptable  health  care  outcomes,  for  avoiding  or  delaying  negative 
outcomes,  and  the  extent  to  which  diagnosis  and  treatment  are  based 
upon  clear  and  widely  accepted  standards  and  criteria. 

Efficacy  of  diagnosis  and  treatment  procedures  is  a  pre- 
requisite for  assuring  the  quality  of  care  in  any  topic  related  category. 
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Table  XXVII 
Factor  III:  Economic  Cost 


TOPIC  RELATED  CATEGORY 

* 

Average  Toal 
Charge  Per 
Discharge 
Exclusive  of 
Physicians' 
Charges, for 
Medicare,  1973 

Median 

Physicians' 

Services 

Charges, 

Maryland 

1973 

Total 

Dollar 
Cost 

Inverse  Rank 
Order 

Component 
Value 

Chronicity 

Modifier 

Follow- Up 

Modifier 

Factor 
Value 

Myocardial  Infarction 

1649 

181 

1830 

14 

7.7 

X 

2 

X 

1 

10.7 

Pneunlonia 

1126 

88 

1214 

9 

5 

0 

0 

5 

Cholecystectomy 

1824 

415 

2239 

19 

10.5 

0 

0 

10.5 

Appendectomy 

1882 

230 

2112 

17.5 

9.6 

0 

0 

9.6 

Hysterectomy 

1386 

475 

1861 

15 

8.3 

0 

0 

8.3 

C-Section 

1662* 

450 

2112 

17.5 

9.6 

X 

2 

0 

11.6 

Tonsillectomy  &  Adenoidectomy 

399* 

118 

517 

2 

1.1 

0 

0 

1.1 

Hip  Fracture 

2401 

450 

2851 

20 

11 

0 

X 

1 

12 

Diabetes 

971 

97 

1060 

6 

3.3 

X 

2 

X 

1 

6.3 

Prostatectomy 

1410 

488 

1898 

16 

8.8 

0 

X 

1 

9.8 

Breast  Lesions 

1428 

208 

1636 

13 

7.2 

X 

2 

X 

'  1 

10.2 

Emergency  Room 

40* 

25* 

65* 

1 

6 

0 

6 

Inguinal  Hernia 

980 

225 

1205 

8 

4.4 

0 

0 

4.4 

Gastroenteritis 

567 

61 

628 

3 

1.7 

0 

0 

1.7 

Ulcer 

919 

84 

1003 

5 

2.8 

X 

2 

X 

1 

5.8 

Cataract 

697 

675 

1372 

11 

6 

0 

X 

1 

7 

Hypertension 

675 

78 

753 

4 

2.2 

X 

2 

X 

1 

5.2 

Delivery 

819* 

325 

1144 

7 

3.9 

X 

2 

0 

5.9 

Cerebrovascular  Accident 

1311 

148 

1459 

12 

6.6 

X 

2 

X 

1 

9.6 

Congestive  Heart  Failure 

1155 

113 

1268 

10 

5.5 

X 

2 

X 

I 

8.5 

Thus,  to  the  extent  that  efficacy  is  found  to  exist  for  outcomes 
and  diagnostic  and  treatment  procedures  for  a  topic  related  category 
the  magnitude  of  problem  rating  assigned  to  that  topic  related  cate- 
gory is  increased.    This  provides  an  index  of  the  potential  for  MCE 
studies  within  the  topic  category  to  contribute  to  improving  the 
health  status  of  the  population. 

Components 

Efficacy  refers  to  the  potential  for  achieving  positive  or 
acceptable  health  care  outcomes  or  for  maintaining  and  assuring  the 
quality  of  care,  within  a  topic  related  category.    The  efficacy  factor 
addresses  the  likelihood  that  treatment  will  be  appropriate  and  benefi- 
cial; that  it  will  reduce  or  postpone  morbidity,  complications,  or 
mortality;  and  conversely  that  poor  treatment  or  no  treatment  will  have 
an  adverse  effect. 

Efficacy  is  measured  by  assigning  a  none/some  rating  to  topic 
categories  for  each  of  the  three  components. 

For  the  20  most  frequently  audited  topic  related  categories, 
these  none /some  ratings  were  made  by  the  Project  Office  in  collaboration 
with  the  Project  Monitor. 

The  three  components  of  the  efficacy  factor  address  diagnosis, 
treatment,  and  outcomes.    These  components  are: 
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a.  Clear  and  accepted  indications  for  treatment  are  said  to 
exist  when  there  is  fundamental  agreement  among  members  of  the  medical 
community  about  the  reliability  of  specific  diagnostic  and  therapeutic 
indicators,  and  when  these  judgments  are  based  upon  established  well 
defined,  operational  criteria.    The  first  component  of  the  efficacy 
factor  is  thus  related  to  diagnosis. 

b.  Clear  and  accepted  modes  of  treatment  are  said  to  exist  when 
there  is  consensus  within  the  medical  community  about  the  appropriate- 
ness of  proposed  interventions. 

c       Likelihood  of  health  benefit  from  treatment.    This  component 
addresses  the  likelihood  that  therapeutic  intervention  will  produce  a 
more  favorable  outcome  than  would  be  obtained  without  such  intervention. 

Factor  Value  Computation 

Factor  values  are  derived  by  combining  component  values  and 
adding  1.    A  positive  component  rating  is  assigned  the  value  of  0.1. 
Thus,  factor  values  for  the  efficacy  factor  may  range  from  1,  express- 
ing identity,  no  increase,  or  lack  of  efficacy,  to  1.3,  expressing  a 
30%  increase  or  a  high  degree  of  efficacy.    These  weights  were  deter- 
mined by  the  Project  Officer,  and  the  MCE  Project  Research  Consultant 
in  collaboration  with  the  Project  Monitor,  all  of  whom  are  physicians. 

As  Table  XXVm  demonstratess ,  when  efficacy  is  measured  for  the 
twenty  most  frequently  audited  topic  related  categories,  one  topic  related 
category  receives  a  low  efficacy  value.    That  is,  efficacy  is  found 
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Table  XXV in 
Factor  IV:  Efficacy 


Clear  and 

Clear  and 

Likelihood 

TOPIC  RELATED  CATFGORY 

Accepted 

Component 

Accepted 

of  Henlth 

(~!  r>  in  rwi  n  r>  n  f 

v.  '*llllJMJ[|l.tH 

I  *  1 1 CUI 1 V  (MlUacf 

Indications 

Value 

Modes  of 

Va  lite 

Bcnef  its 

Vfilu© 

Modi  f  1  f»r 

for  Treatment 

Treatment 

Myocardial  Infarction 

X 

.  1 

0 

0 

0 

.  0 

1.1 

Pneumonia 

X 

.  1 

X 

.1 

X 

.  l 

1.3 

Cholecystectomy  , 

X 

.  1 

X 

.1 

X 

.1 

1.3 

Appendectomy  \ 

X 

.  1 

X 

.1 

X 

.1 

1.3 

Hysterectomy  • 

0 

0 

X 

.  1 

X 

• .  l 

1.2 

C- Sect  ion 

0 

0 

X 

.  1 

X 

.1 

1.2 

Tonsillectomy  &  Adenomectomy 

0 

0 

X 

.  1 

0 

0 

1.1 

Hip  Fracture 

X 

.  1 

X 

.  1 

X 

.  1 

1.3 

Diabetes 

0 

0 

0 

0 

0 

0 

1.0 

Prostatectomy 

X 

X 

.  1 

X 

.1 

1.3 

Breast  Lesions 

X 

0 

0 

X 

.  1 

1.2 

Emergency  Room 

0 

X 

.  1 

1.2 

Inguinal  Hernia 

X 

X 

.  1 

X 

.1 

1.3 

Gastroenteritis 

X 

0 

0 

0 

0 

1.1 

Ulcer 

X 

0 

0 

0 

0 

1.  1 

Cataract 

X 

X 

.  1 

X 

.1 

1.3 

Hypertension 

X 

X 

.  1 

X 

.  1 

1.3 

Delivery 

X 

X 

.1 

X 

.1 

1.3 

Cerebrovascular  Accident 

X 

0 

0 

0 

0 

1.1 

Congestive  Heart  Failure 

X 

X 

.1 

X 

.  1 

1.3 

to  be  lacking  for  diabetes. 

Computing  the  Relative  Chance  of  Finding  and  Removing 
Randomly  or  Systematically  Distributed  Problem  Areas 

When  factor  values  are  computed  for  all  four  factors,  a 
relative  potential  impact  ranking  may  be  derived  for  the  topic  related 
categories  under  consideration. 

This  procedure  involves  two  steps: 

1)  Multiply  the  sum  of  Factors  I,  I,  and  II  by  the 
efficacy  factor  value  (Factor  IV)  for  each  topic 
category.    This  will  yield  a  potential  impact 
rating. 

2)  Arrange  the  potential  impact  ratings  in  rank  order. 
This  will  yield  a  relative  potential  impact  ranking. 

This  procedured  was  applied  to  the  20  most  frequently  audited 

topic  related  categories  and  the  results  are  presented  in  columns  six 

and  seven  of  Table  XXK.     Of  these  topic  related  categories,  breast 

lesions  and  appendectomy  were  found  to  have  the  greatest  potential  for 

contributing  to  the  health  status  of  the  population.  Cholecystectomy, 

hypertension,  pneumonia,  congestive  heart  failure,  cerebrovascular 

accident,  myocardial  infarction,  delivery  and  hysterectomy  were  also 

highly  ranked. 

The  first  three  factors  were  added  in  the  probability  sense 
that  either  I,  II,  or  III  may  be  large  enough  to  ensure  the  discovery 
of  a  randomly  or  systematically  distributed  problem  area.  However, 
the  Factor  IV  is  multiplied  in  the  probability  sense  that  a  category 
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Table  XXIX 


Chance  of  Finding  Problem  Areas  in  the  Twenty  Most 
Frequently  Audited  Topic  Categories  (Criterion  1) 


TOPIC  RFT .ATFD  TATFCDRV 

■ 

Pacient  Value 

Provider  Value 

Economic  Value 

Magnitude  of  Problem 
Rating 

Efficacy  Modifier 

Potential  Impact 
Rating 

Relative  Potential 
Impact  Ranking 

Myocardial  Infarction. 

5.8 

10.2 

10.7 

2S.7 

1.1 

29.4 

8 

Pnuemonia 

11.2 

9.0 

5 

25.2  . 

1.3 

32.8 

4.5 

Cholecvst  ecf  omv 

9.1 

7.2 

10.5 

26.8 

1.3 

34.8 

3 

Appendectomy 

9.8 

7.8 

9.  6 

27.2 

1.3 

35.4 

1.5 

Hysterectomy 

11.9 

3.5 

8.3 

23.7 

1.2 

23.4 

10 

C-Section 

2.8 

3.5 

11.6 

17.9 

1.2 

21.5 

17 

Tonsilleccomy  &  Adenoidectomy 

12.6 

6.4 

1.1 

20.1 

1.1 

22:.  I 

15 

Hip  Fracture 

2.1 

6.6 

12 

20.7 

1.3 

26.9 

12.5 

Diabetes 

'  6.9 

13.7 

6.3 

26.9  • 

1.0 

26.9 

12.5 

Prostatectomy 

!  6.3 

4.C 

9.8 

20.1 

1.3 

26.1 

14 

Breast  Lesions 

10.5 

8.8 

10.2 

29.5 

1.  2 

35.4 

1.5 

Emergency  Room 

14 

2.2 

.6 

16,8 

1.2 

20.2  ■ 

18 

Inguinal  Hernia 

7 

5.4 

4.4 

16,8 

1.3 

21.8 

16 

Gastroenteritis 

.7 

7.6 

1.7 

10 

1.1 

11.7 

20 

Ulcer 

7.9 

10.8 

5.8 

24^5 

1.1 

27  ■ 

11 

Cataract 

3.5 

1.5 

7 

12 

1.3 

15.6 

19 

Hypertension 

6.8 

13.2 

5.2 

25.2 

1.3 

32.8 

4.5 

Delivery 

13.3 

2.9 

5.9 

22.1 

1.3 

28.7 

9 

Cerebrovascular  Accident 

6 

11.3 

9.6 

27.4 

1.1 

30.1 

7 

Congestive  Heart  Failure 

4.9 

11.4 

8.5 

24.8 

1.3 

32.2 

6 
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must  be  related  to  known  benefits  which  can  be  realized  by  interven- 
tion (MCE  and  otherwise). 

5.4  Criterion  2:    Situation  Specific  Factor:  Empiric  Rating  of 

PSRO  Methods  Used  in  Selecting  Topics 

An  MCE  topic  which  belongs  to  a  category  with  low  rated  impact 

potential  according  to  Factors  I- IV  need  not  have  a  low  overall  rating 

if  it  is  selected  because  an  audit  committee  wishes  to  examine  an  area 

it  feels  is  problematic.    As  the  Institute  of  Medicine  points  out:  "A 

targeted  review  should  enable  more  frequent  identification  of  truly 

inappropriate  care  and  less  frequent  review  of  appropriate  care,  which 

should  increase  the  efficiency  of  review."5    Thus,  even  if  the  random 

chance  of  finding  problem  areas  is  low  due  to 'a  small  scope  of  the 

topic  category,  the  overall  chance  of  finding  problems  may  be  high  when 

the  review  is  appropriately  targeted. 

Procedure 

A  10%  random  sample  of  BQA  131  reporting  forms  was  drawn  from 
all  studies  in  the  twenty  most  frequently  audited  topic  related  cate- 
gories in  the  sample  data  base.    This  sample  consisted  of  158  studies. 
The. PSRO  reporting  each  study  was  telephoned  to  request  additional 
documentation  from  the  originating  hospital  concerning  the  reasons 
for  individual  study  topic  selection.    The  PSROs  indicated  that  they 
encountered  some  difficulty  in  obtaining  this  data  because  of  the  short 
turn  around  time  involved.    Nevertheless,  additional  information  was 
obtained  for  108  of  the  158  studies  or  68%.    These  108  studies  were 
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then  rated  acceptable  or  unacceptable  in  terms  of  topic  selection 

by  two  independent  raters,  a  physician  and  a  public  health  specialist. 

Agreement  was  obtained  in  each. 

Rating  Criteria 

The  individual  topics  were  rated  as  acceptable  from  the  situa- 
tion specific  standpoint  if  reasons  were  cited  as  to  a  local  situation 
in  the  hosptial  which  gave  particular  importance  to  such  a  study  topic 
at  that  time.    Among  the  reasons  cited  were: 

«     Unusually  high  frequency  for  a  given  disease 
or  treatment  procedure. 

•  Hospital  concern  about,  a  particular  problem  which 
was  suspected  to  be  present  or  had  been  revealed 

by  concurrent  review,  profile  studies  or  death  con- 
ferences. 

•  A  desire  to  study  a  topic  which  was  the  subject  of 
current  research  at  the  hospital. 

•  Physician  concern  that  criteria  for  a  particular 
topic  area  were  inadequate  and  needed  clarification. 

A  none/some  determination  was  used  and  a  liberal  rating  standard 

was  applied  in  order  to  come  up  with  a  reasonable  spread  between  the  two 

categories. 

Frequency  was  by  far  the  most  often  cited  reason  for  topic 
selection.    Audit  notes- rarely  reflected  the  level  of  investigation 
or  thought  on  the  part  of  the  audit  committee  concerning  other  rea- 
sons for  selecting  the  topic,  such  as  its  particular  relevance  to  the 
health  status  of  the  population  being  served  by  the  hospital  or 
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concern  about  appropriateness  of  diagnosis,  treatment  conditions 
or  administrative  issues. 

For  judging  individual  topic  selection  in  the  future,  cri- 
teria should  be  applied  more  rigorously.    This  survey  indicates  that 
further  guidance  from  BQA  or  technical  assistance  may  be  indicated 
regarding  methods  for  selecting  health  care  priority  topics  within 
an  institution. 

Factor  Value  Computation 

In  view  of  the  paucity  of  data  available,  no  attempt  was 
made  to  divide  this  factor  into  component's  with  individual  value  rat- 
ings and  a  minimal  score  for  acceptability.    Any  reasonable  local 
cause  for  selection  was  deemed  sufficient  to  classify  the  topic  as 
accpetable  for  situation  specific  reasons. 

For  the  purpose  of  arriving  at  a  modifier  to  be  applied  to 
the  potential  impact  ranking  of  the  topic  categories  being  assessed, 
the  percentage  of  studies  in  each  category  found  acceptable  was  used  as 
the  weighted  value.    The  computation  is  shown  in  TableXXX.  The  total 
number  of  audits  reviewed  for  each  category  is  shown  in  column  one. 
Those  for  which  no  additional  data  was  obtained  are  shown  in  column 
two. 

Those  for  which  the  additional  data  still  did  not  permit 
an  acceptable  rating  for  situation  specific  reasons  are  shown  in 
column  three.    Additional  data  which  did  permit  an  acceptable  rating 
for  the  individual  topic  is  shown  in  column  four.    Column  five  present; 
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Tabe  XXX  . 
Factor V:  Situation  Specific  Rankings 


TOPIC  CATEGORY 

Total 
Studies 
Reviewed 

No 
Data 
Obtained 

Number  Con- 
sidered Not 
Acceptable 

i  Number 
Considered 
Accents  hip 

%of 
Sample 

Rank 
Order 

Factor 
Value 

Myocardial  Infarction 

16 

6 

5 

5 

31 

16 

0.6 

Pneumonia 

14 

4 

5 

5 

3G 

14 

0.7 

Cholecystectomy 

12 

6 

1 

5 

42 

9 

0.8 

Appendectomy 

11 

7 

1 

3 

27 

19 

0.5 

Hysterectomy 

9 

1 

2 

6 

G7 

5 

1.3 

C-Section 

8 

2 

0 

G 

75 

3 

1.5 

Tonsillectomy  &  Adenomectomy 

8 

2 

3 

,  3 

38 

12.5 

0.8 

Hip  Fracture 

8 

1 

4 

3 

38 

12.5 

0.8 

Diabetes 

8 

3 

1 

4 

50 

8 

1.0 

Prostatectomy 

7 

4 

1 

2 

29 

17.5 

0.6 

Breast  Lesions 

7 

3 

2 

2 

29 

17.5 

0.6 

Emergency  Room 

6 

2 

0 

4 

G7 

5 

1.3 

1  nfi*i  i  in  n  1  Ifpy'iin 

£1  111  11  111  ll  1  llt~i.lll.ct 

a 

u 

i 

X. 

i 

A 

4 

1X1 
U  1 

D 

i  i 
1. .  o 

Gastroenteritis 

G 

0 

1 

5 

83 

1 

1.7 

Ulcer 

6 

2 

3 

1 

17 

20 

0.3 

Cataract 

G 

2 

2 

2 

33 

15 

0.7 

Hypertension 

5 

1 

1 

3 

GO 

7 

1.2" 

Delivery 

5 

0 

1 

4 

80 

2 

1.6 

Cerebrovascular  Accident 

5 

1 

2 

2 

40 

10.5 

0.8 

Congestive  Heart  Failure 

5 

3 

0 

2 

40 

10.  5 

0.8 

Total 

158 

50 

37 

71 

I 


the  number  rated  acceptable  as  a  percentage  of  the  total  data  base 
for  each  category.    Column  six  presents  the  ranking  of  each  category 
based  upon  this  percentage  of  studies  within  the  category  which  were 
found  to  have  acceptable  situation  specific  considerations. 

Situation  specific  factors  can  have  a  major  impact  on  an 
individual  study's  entrance  into  the  decision  tree  concerning  acceptability 
as  explained  in  Section  5.2  (p.120)  and" are  therefore  a  major  considera- 
tion in  rating  individual  studies.    However,  for  the  purpose  of  the 
current  study  in  arriving  at  a  selection  of  topic  categories  with  a  high 
concentration  of  valid  topics,  it  was  decided  that  specif icicty  chare- 
teristics  of  the  sample  studied  should  not  be  given  major  weight  in 
modifying  the  potential  impact  ranking  previously  determined  by  the 
process  described  in  Section  5.3.    This  was  particularly  true  in  view 
of  the  liberal  criteria  used  for  judging  situation  specificity  for  this 
sample.    A  total  weight  of  0.02  was  therefore  assigned  to  the  situation 
specificity  factor.    This  was  multiplied  by  the  percentage  of  studies 
found  acceptable  in  each  category  to  determine  the  factor  value  as 
shown  in  column  7. 

5.5  Criterion:3Topic  Specificity  Factor:  Empiric  Rating  of 

-Methods  in  Specifying  Specific  Topics 

For  an  individual  MCE  study  to  be  considered  valid,  the  topic 

it  addresses  usually  should  be  sufficiently  specific  and  focused  to  permit 

problem  areas  in  medical  care  within  that  topic  to  be  highlighted.  The 
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rating  of  an  individual  study  topic  can  only  be  made  by  examining 
its  title  as  reported  on  PMIS  form  BQA  131  agianst  criteria  establish- 
ing limits  to  the  size  of  patient  and  provider  population  and  the  scope 
of  the  medical  condition  or  hospital  procedure  which  can  reasonably  be 
investigated  in  one  study.    Although  strict  evaluation  of  the  specificity 
of  individual  MCE  study  topics  is  considered  desirable,  until  there  is 
general  consensus  and  acceptance  by  participating  hospitals  of  the 
importance  of  selecting  a  sufficiently  focused  and  manageable  study 
topic  and  criteria  to  be  used  in  doing  so,  it  would  be  unrealistic  to 
attempt  to  develop  a  more  detailed  rating  system  for  topic  specificity. 

This  section,  therefore,  presents  a  method  for  assessing 
specificity  of  individual  MCE  topics  according  to  a  liberal,  rudimen- 
tary procedure  and  for  deriving  a  factor  value  for  the  proportion  of 
such  studies  in  a  topic  category  which  are  found  acceptable.    This  may  be 
used  to  modify  category  rankings  determined  previously.    One  should  note 
that  such  a  modifier  applies  only  to  the  sample  being  analyzed  and  is  only 
significant  for  the  purpose  of  selection  of  studies  for  further  in-depth 
analys. 

Procedure 

A  25%  random  sample  of  BQA  131  reporting  forms  was  drawn 
from  the  total  universe  of  MCE  audits  for  each  of  the  twenty  most  fre- 
quently audited  topic  categories.    An- acceptable/unacceptable  (none/some) 
judgment  was  made  by  the  Project  Manager,  an  experienced  health  services 
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evaluator,  concerning  each  of  these  395  individual  study  topics  as 
to  whether  that  topic  had  been  described  in  sufficiently  specific 
terms  to  be  useful  for  defining  the  study  objectives  and  parameters. 
These  judgments  were  reviewed  and  comfirmed  by  the  MCE  Project  Con- 
sultant, a  physician.    The  percentage  of  studies  in  each  topic  related 
category  judged  to  have  acceptable  specificity  was  then  used  to 
establish  a  specificity  ranking  among  the  categories.    This  percentage 
figure  for  positive  specificity  for  each  topic  related  category  was 
used  to  determine  a  ranking  modifier  for  use  in  combining  the  results 
of  the  topic  specificity  analysis  with  those  made  for  the  potential 
impact  and  situation  specific  analyses. 

The  result  of  this  procedure  is  presented  in  Table  XXXI. 
The  total  number  of  audits  reviewed  for  each  topic  category  is  shown 
in  column  one.    The  number  unacceptable  are  listed  in  column  two;  this 
is  shown  as  a  percentage  of  the  studies  in  the  category  in  column  three. 
The  number  found  acceptable  is  listed  in  column  four;  the  percentage 
acceptable  in  column  five.    The  rank  order  of  the  topic  related  cate- 
gories based  upon  this  percentage  of  acceptable  studies  is  shown  in 
column  six.    The  factor  value  is  listed  in  column  seven. 

Ranking  Criteria 

The  criteria  employed' to  judge  the  present  sample  were  liberal. 
Two  types  were  employed:  modifiers  or  descriptors  in  the  topic  subject 
limiting  the  size  of  patient  population  and/or  the  scope  of  the  disease 
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Table  XXXI 
factor  VI:  Topic  Specificity  Rankings 


Total 

Number 

% 

N  u  m  be  r 

Rank 

Factor 

Studies 

Unaccept- 

Unacceptable 

Acceptable 

Acceptable 

Order 

Value 

TOPIC  CATEGORY 

Reviewed 

able 

Myocardial  Infarction 

39 

35 

90 

4 

10 

17.5 

0.3 

Pneumonia 

34 

11 

32 

23 

68 

a 

2.0 

Cholecystectomy 

31 

25 

81 

6 

19 

15 

0.6 

Appendectomy 

28 

15 

54 

13 

46 

6 

1  .4 

Hysterectomy 

25 

17 

68 

8 

32 

11 

1.0 

C-Seclion  v 

21 

9 

43 

12 

57 

5 

1.7 

Tonsillectomy  *r  AdCnoidectomy 

20 

18 

90 

2 

10 

17.5 

0.3 

Hip  Fracture 

on 

7 

00 

O  A 
£.  .  If 

Diabetes 

20 

15 

75 

5 

25 

125 

0.8 

Pmstatectomy 

10 

18 

95 

1 

5 

20 

0.2 

Breast  Lesions 

16 

10 

62 . 5 

6 

37 . 5 

7 

1 . 1 

Emergency  Room 

If. 

2 

12.5 

14 

88.5 

I 

2.7 

Inguinal  Hernln 

15 

14 

93 

1 

7 

19 

0.2 

G  astro  enter)' Is 

14 

4 

29 

10 

71 

2 

2.1 

Ulcer 

14 

9 

64 

5 

36 

9 

1.1 

Cataract 

14 

9 

64 

5 

36 

9 

l.l 

Hypertension 

14 

11 

79 

3 

21 

14 

0.0 

Delivery 

11 

7 

64 

'  4 

36 

9 

1.1 

Cerebrovascular  Accident 

12 

9 

75 

3 

25 

125 

0.8 

Congestive  Heart  Failure 

12 

10 

83 

2 

17 

16 

0.5 

Total 

395 

255 

65 

140 

35 

studied.  This  was  advisable  because  only  a  dozen  or  so  of  the  topics  reported  on 
the  Form  131  provided  what  one  might  expect  for  a  full  and  complete  title; 
only  five  contained  study  objectives.    The  use  of  more  demanding  criteria 
at  this  time  would  have  made  it  impossible  to  obtain  comparisons  among  the 
topic  categories.    More  stringent  criteria  should  be  employed 
in  the  future.    Technical  assistance  may  be  required  to  improve  report- 
ing requirements  and/or  to  assist  in  topic  definition. 

Although  the  data  obtained  concerning  overall  topic  specificity, 
even  using  this  liberal  criterion  were  disappointing,  one  should  note  that 
a  non-specific  topic  does  not  necessarily  mean  a  poor  study.    A  specific 
topic  assists  examiners  in  understanding  the  objectives  of  a  study.  Non- 
specificity  can  be  overcome,  however,  by  detailed  development  of  objec- 
tives, methods  and  criteria. 

Factor  Value  Computation 

Given  the  constraints  noted  above,  any  specificity  modified 
applied  to  the  sample  under  consideration  should  be  given  less  weight  than 
might  be  desirable.    A  value  ratio  of  0.03  was  selected  and  applied  to 
the  percentage  of  studies  in  each  category.    The  resulting  topic  speci- 
ficity factor  value  is  shown  in  column  7. 

5.6  Relative  Topic  Category  Probability  of  Containing  MCEs 

With  Impact 

When  two  MCEs  are  chosen  for  evaluation,  their  relative 
probability  of  identifying  problem  areas  is  based  on  the  overall  relative 
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ranking  of  their  Topic  Categories  according  to  the  three  criteria 
above.    The  Topic  Category  with  the  highest  ranking  is  one: 

1.  Whose  scope  is  largest  and  whose  disease  related 
benefits  are  well  known.    This  maximizes  the 
random  chance  of  finding  removable  problem  areas 
at  initial  audit. 

2.  Whose  random  sample  of  MCEs  had  the  best  situation 
specific  rationales  for  selecting  a  topic. 

3.  Whose  random  sample  of  MCEs  had  best  focused  their 
topics.    Combinations  of  the  three  criteria  yield 

an  overall  ranking  of  the  Topic  Category  probability 
of  containing  MCEs  with  potential  impact. 

The  final  topic  category  score  was  generated  by  adding  the 
factor  values  described  in  earlier  sections  for  each  of  the  criteria. 

As  is  seen  on  Table xxxifon  the  next  page,  .'MCEs  in  the  cate- 
gories of  myocardial  infarction,  pneumonia,  cholecystectomy,  appendec- 
tomy, breast  lesion,  hypertension,  delivery,  CVA,  and  CHF  would  have 
an  above  average  probability  of  having  MCEs  with  impact. 
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Table  XXXn 

SAMPLE  VALIDITY  RANKING 


TOPIC  RELATED  CATEGORY 

T?  pi  o  f  i  trp 

"Pnf  pnf  i  ^1 
T"rnnn  r^f 

Ranking 

Rif  nof  i  am 
Sttip/**  i"f  i  /*» 

Ranki  no* 

Trvrn  r* 
J.  <jjjic 

T?   nlci  no* 

"Pi  nal 

±  max 
uaxnpie 

"\fo1  i  r*  ^  i.  t  r 

v  cLiiuiry 

T?  p  nl/"?  n  cr 

lldLllVLU.^ 

Myocardial  Infarction 

8 

16 

17.5 

9 

Pneumonia 

4.5 

14 

3 

4 

Cholecystectomy 

3 

9 

15 

3 

Appendectomy 

1.5 

19 

c 

6 

1 

Hysterectomy 

10 

5 

11 

10 

C-Section 

17 

3 

5 

15 

Tonsillectomy  &  Adenoidectomy 

15 

12.5 

17.5 

16 

Hip  Fracture 

12.5 

12.5 

4 

12 

Diabetes 

12.5 

8 

12.5 

13 

Prostatectomy 

14 

17.5 

20 

14 

Breast  Lesions 

1.5 

17.5 

7 

2 

Emergency  Room 

18 

5 

1 

17 

Inguinal  Hernia 

16 

5 

19 

18 

Gastroenteritis 

20 

1 

2 

19 

Ulcer 

11 

20 

9 

11 

Cataract 

19 

15 

9 

20 

Hypertension 

4.5 

7 

14 

5 

Delivery 

9 

2 

9 

8 

Cerebrovascular  Accident 

7 

10.5 

12.5 

7 

Congestive  Heart  Failure 

6 

10.5 

16 

6 
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5.7 


Follow-Up  Procedures  -  Implementation  of  Follow-Up  In 
The  MCE  Program   


The  MCE  Program  truly  may  have  an  impact  on  medical  care 
process  and  patient  outcomes  only  if  problem  areas,  once  identified, 
are  removed  by  the  implementation  of  PSRO  follow-up  procedures.  In 
assessing  this  overall  implementation  of  follow-up,  we  are  again  eval- 
uating the  potential  impact  of  the  MCE  Program.    Follow-up,  including 
the  intervention  of  recommended  actions  and  the  completion  of  reaudits, 
are  assessed  here  in  two  sub- reports. 


5.7.1  Assessment  of  Total  Follow-up 

in  a  10%  Random  Sample  of  MCE 

5.7.2  Overall  MCE  Reaudit  Activity  in 

the  MCE  Program 


5-7.1        Assessment  of  Total  Fol low-Up  Performed  in  a  10%  Random 
Sample  of  MCEs  ~~ 

Methods 

To  access  the  level  and  potential  impact  of  follow-up  activity 
being  performed  in  the  context  of  medical  care  evaluation  studies,  a  ran- 
dom sample  of  these  studies  was  drawn.    This  sample  included  10%  of  the 
MCE  studies  begun  before  January  1976  by  the  first  65  PSROs  to  reach 
conditional  status.    Eighty  studies  were  included  in  this  sample.  However, 
three  of  these  studies  were  deleted  because  documentation  could  not  be 
obtained  in  time  for  the  analysis. 

For  the  remaining  sample  base  of  77  studies,  BQA  forms  131  and 
133  were  collected  where  available.    Information  on  these  forms,  "Medical 
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Care  Evaluation  Study  Abstract"  and  "Restudy  Report,"  was  supplemented 
by  data  gathered  by  a  telephone  survey  of  the  PSROs  involved.  Each 
PSRO  was  asked  detailed  questions  concerning  the  follow-up  aspects  of 
every  study  within  the  sample  which  was  conducted  under  its  jurisdic- 
tion. 

All  available  information  was  collected  and  aggregated  into 
the  four  major  diagnostic  conditions  specified  by  the  proposal  for  the 
current  contract: 


•  Condition  I: 
t  Condition  II: 

•  Condition  III; 

•  Condition  IV: 


Interventions  not  required 

.Interventions  required,  inter- 
ventions not  carried  out 

Interventions  required,  inter- 
ventions carried  out,  reaudit 
not  performed 

Interventions  required,  inter- 
ventions carried  out,  reaudit 
performed 


Overall  Status  of  Current  MCE„FoT low-Up  Activity 
Data  from  the  10%  random  sample  of  MCE  studies  begun  before 
January,  1976  is  presented  in  this  section.    Statistical  analyses  of 
this  information  have  been  performed  for  each  of  the  seven  conditions, 
for  some  sub-conditions,  as  well  as  across  all  conditions, 
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The  classification  characteristics  of  the  sample  population 
•are  indicated  on  Table  XXXm. 


Table  XXXm 

DISTRIBUTION  OF  AUDITS  IN  10%  SAMPLE, 
BY  CLASSIFICATION 


Classification 

Number  " 

% 

A -Medical 

36 

47 

B-Surgical 

31 

40 

C  -  A  dmi  ni  st  rat  i  v  e 

10 

13 

TOTAL 

77 

100% 

The  distribution  reflects  the  overall  distribution  of  2,876 
MCE  initial  audits  studies  in  the  data  base  at  the  time  of  this  study. 

The  overall  level  and  type  of  follow-up  activity  being  per- 
formed in  the  context 'of -MCE  studies  is  indicated  by  Table  XXXV,  on 

the  following  page. 


-157- 


Table  XXXV 

STATUS  OF  CURRENT  MCE  FOLLOW-UP  ACTIVITY 


v^onuiuicn 

Total  in 
v--unuiiiuii 

Percent  in 
^-u  ii  '-Li  -  iun 

I.     Interventions  not  required 

26 

34 

II.    Interventions  required,  interventions  not 
pirripd  out 

2 

3 

III.   Interventions  required,  interventions  carried 
out,  re-audit  not  performed 

3G 

47 

IV.     Interventions  required,  interventions  carried 
out,  rc-audit  peiiormed 

13 

17 

TOTAL 

77 

101%* 

♦difference  due  to  rounding  error 


Table  XXXV  indicates  that  follow-up  interventions  assessed  and 
and  reported  by  hospitals  are  not  required  in  34%  of  all  MCE  studies 
in  this  sample.    The  remaining  66%  of  the  studies  identify  deficiencies 
which  indicate  the  need  for  interventions. 

Of  the  group  which  requires  interventions,  recommended  actions 
were  carried  out  in'aj}  but  two  instances  or  96%  of  the  time.  However, 
only  13  reaudits  were  conducted  for  the  group  of  51  initial  audits 
identifying  deficient  practices.    In  other  words,  only  25%  of  those 
sample  audits  identifying  deficiencies  led  to  a  reaudit  within  the 
one-year  period  allotted  by  PSRO  reporting  guidelines.    Another  group 
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of  7  studies  identifying  deficiencies  reported  audits  in  progress. 

Thus,  39%  of  all  audits  identifying  deficient  hospital  practicess  resulted 

in  reaudits  completed  or_in  progress. 

In  two  instances  recommended  interventions  were  not  carried 
out,  and  in  six  other  instances  recommended  interventions  were  not 
completely  carried  out.    Finally  *  two  reaudits  were  performed  for 
audits  which  did  not  originally  identify  deficiencies  or  recommend 
corrective  follow-up  interventions. 

Status  of  Follow-up  Implementation,  by  Condition 

1 .     Condition  I:    Interventions  Not  Required 

c 

Table  XXXVI 
MCE  AUDITS  NOT  REQUIRING  INTERVENTIONS 


Explanation 

Number 

Percent  in 
Condition 

No  Deficiency 

n 

42 

Deficiency  Justified  ' 

15 

58 

TOTAL 

26 

100% 

As  Table  XXXVI  demonstrates,  26  audits  did  not  recommend  inter- 
ventions.    This  represents  34%  of  the  studies  sampled.    Of  this  group 
42%  of  the  audits  reviewed  hospital  care  situations  in  which  no  defi- 
ciency was  identified.    In  the  remaining  58%  of  the  audits,  the 
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deficiencies  which  were  identified  were  considered  justified  by  the 
individuals  responsible  for  the  audit. 

2.  Condition  II:.  Interventions  Required,__Interventions 
Not  Carried  Out 

Only  two  audits  recommended  interventions  which  were  not 

carried  out.    This  was  justified  in  one  instance.    In  the  first 

instance  the  recommended  actions  could  not  be  carried  out  because 

the  two  doctors  in  question  both  left  the  hospital.    In  the  second 

instance  it  was  recommended  that  a  staff  meeting  be  held  to  address 

the  problem.    There  is  no  indication  as  to  why  this  was  not  done. 

3.  Condition  III:  Interventions  Required,  Interventions 
Carried  Out,  Reaudit  Not  Performed 

The  largest  number  of  audits  in  this  sample  fall  into 

Condition  III.    For  47%  of  the  total  sample  group,  interventions  were 

recommended  and  carried  out  yet  no  reaudit  was  performed.  Though 

reaudit  is  not  strictly  required  by  PSRO  guidelines  if  it  is  not 

indicated  by  the  appearance  of  unjustified  deficiencies  in  health 

care  practices,  in  most  instances  where  interventions  are  recommended 

a  reaudit  should  be  performed.    Thus,  the  magnitude  of  the  sample 

group  in  Condition  ML  may  indicate  a  weakness  in  the  implementation 

of  the  full  MCE  study  cycle. 

4.  Condition  IV: .Interventions  Required,  Interventions 
Performed,  ReaudU  parried  Out 

Condition  IV  is  the  MCE  follow-up  condition  which  most 

convincingly  assures  improvement  in  the  quality  of  hospital  care. 
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In  this  condition  required  interventions  in  deficient  medical  care 
situations  have  been  made,  and  the  impact  of  these  interventions 
has  been  demonstrated  by  reaudit.    Only  17%  of  the  MCE  studies  in 
the  total  sample  fall  within  this  condition,  although  the  sample  was 
chosen  in  such  a  way  that  the  audits  recommending  interventions  would 
have  the  requisite  one  year  period  for  completing  a  reaudit. 

Reaudits  should  demonstrate  improved  quality  of  care.  The 
actual  impact  of  corrective  follow-up  interventions  on  deficiencies 
in  reaudited  health  care  situations  is  displayed  on  Table  XXXVE. 

Table  XXXVII e 

INTERVENTIONS  REQUIRED,  INTERVENTIONS  CARRIED  OUT 
RE-AUDIT  PERFORMED,  BY  OUTCOME 


Outcome 

Number 

% 

Classification 

All  deficiencies  eliminated,  no 
new  deficiencies  found 

5 

38 

A  =3 
B=l 
C=l 

All  deficiencies  eliminated,  1-4 
new  deficiencies  found 

6 

46 

A=4 
B=2 

Deficiencies  reduced  but  not  elimi- 
nated, no  new  deficiencies  found 

2 

15 

C=2 

TOTAL 

13 

99*% 

♦Difference  due  to  rounding  error 
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5.7.2       Overall  MCE  Reaudit  Activity  in  the  Nation 

The  above  analysis  of  follow-up  procedures  was  based  on 
a  random  sampling  of  MCEs.    The  following  information  is  a  nation- 
wide summary  of  reaudit  information  in  the  PMIS  data  bank.    It  is 
presented  for  completeness.    It  appears  that  although  the  reaudit 
is  a  critical  component  of  the  entire  MCE  study  cycle,  it  has  not 
been  implemented  at  the  same  rate  as  the  PSROs'  concurrent  review  and 
initial  audit  functions.    Whereas  2,876  MCE  initial  audits  have  been 
reported  as  completed  by  June  1,  1977,  only  132  reaudits  were  reported 

as  completed  by  this  time . 

Reaudits  usually,  though  not  always,  are  indicated  where  a 

deficiency  in  health  care  administration,  delivery  or  patient  outcomes 
is  identified  by  an  initial  MCE  audit.    Approximately  69%  of  all  initial 
audits  indicate  the  necessity  of  a  reaudit.    Thus,  reaudits  are  recom- 
mended for  43.5%  of  all  initial  audits,  and  56.5%  of  all  initial  audits 
indicate  that  no  reaudits  are  necessary.    Consequently,  of  the  2,876 
initial  audits  reported  by  Quarter  2,  1977,  one  would  expect  1,251 
to  lead  to  reaudits  and  the  remaining  1,625  to  be  complete  after  termi- 
nation of  the  initial  audit  and/or  intervention. 

Thus,  approximately  one  reaudit  has  been  completed  for  every 
22  initial  MCE  audits'/and  for  every  three  initial  MCE  audits,  one 
reaudit  is  planned  and  may  be  in  process  (assuming  that  all  planned 
reaudits  will  be  carried  out).    This  represents  at  best  a  33.3%  rate 
of  reaudit  activity,  whereas  one  could  expect  a  rate  as  high  as  69% 

y    The  number  is  greater  than  132.  PSRO  errors  in  reporting 
and  PMIS  delays  are  such  that  one  PSRO  has  done  90  reaudits 
even  though  only  20  were  acceptable  for  entry  into  the  PMIS 
data  base  at  the  time  of  this  study. 
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if  a  reaudit  were  performed  whenever  a  deficiency  was  found  during 
an  initial  MCE  audit,  or  as  high  as  43.5%  if  reaudits  were  only 
performed  for  those  deficiency-identifying  audits  which  indicate  that 
a  reaudit  is  necessary.    Thus,  rather  than  1,096  reaudits  reported 
planned  or  completed  one  would  expect  between  1,259  and  1,999. 

Of  the  1  ,096  reaudits  reported  as  planned  or  completed,  132 
reaudits  were  reported  as  complete  on  status  reports  (BQA  from  135), 
another  964  reaudits  were  reported  to  be  planned  or  in  process  as 
of  Quarter  2,  1977,    There  is,  however,  evidence  to  suggest  that  a 
substantial  number  of  planned  reaudits  are  not  being  carried  out. 
Telephone  interviews  were  conducted  with  PSROs  on  a  20%  random  sample 
of  reaudits  that  were  planned  to  have  been  started  by  the  end  of 
Quarter  1,  1977.    Of  the  36  planned  reaudits  in  the  sample,  only  13 
reaudits  were  completed  or  in  process.    Of  the  remaining  23  anti- 
cipated reaudits  14  had  not  been  started,  three  had  been  discontinued 
because  there  was  an  insufficient  number  of  cases  for  reaudit,  and 
six  had  been  abandoned  for  various  other  reasons.    Thus,  of  the 
sampled  audits  which  were  reportedly  to  commence  in  Quarter  1,  1977, 
only  36%  were  completed  or  in  process  as  of  the  first  week  of  June, 
1977. 

Similar  results  were  obtained  when  the  same  procedure  was 
repeated  for  reaudits  which  were  reportedly  to  begin  by  the  end  of 
Quarter  2,  1977.    In  this  20%  sample  of  a  universe  of  223  planned 
reaudits,  only  40%  of  the  reaudits  were  completed  or  in  process. 
The  remainder  had  not  been  started. 
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In  summary,  these  surveys  of  PSROs  which  indicated  that 
reaudits  had  been  planned  suggest  that  approximately  62%  of  such 
planned  reaudits  have  not  begun  on  schedule.    One  may  therefore 
infer  that  the  figures  of  964  reaudits  which  were  reported  planned 
to  be  in  process  by  this  time  may  overstate  the  reality  of  reaudit 
activity.    If  these  samples  are  accepted  as  representative,  it  is 
probable  that  only  approximately  366  of  these  reported  reaudits  are 
being  carried  out  on  schedule. 

The  relatively  low  level  of  reaudit  activity  may  be  a  result 
of  several  factors.    These  factors  include  the  recency  with  which 
initial  audits  were  implemented  and  the  resulting  lag  time  for  reaudit; 
administrative  issues  at  the  PSRO  level;  lack  of  incentives  for  per- 
forming the  reaudits;  and  resistance  to  medical  audit,  the  educational 
intervention  process,  or  the  PSRO  program  within  the  medical  profession. 

Increase  in  MCE  Reaudit  Activity 

There  is,  in  spite  of  these  actual  or  potential  problems, 
a  noticeable  trend  towards  growth  in  the  number  of  completed  reaudits 
on  a  quarterly  basis.    As  Chart  V  shows,  the  first  six  reaudits  were 
completed  in  Quarter  4,  1975,  and  the  following  four  quarters  saw 
the  completion  of  8,"-27  ,  39  and  42  reaudits,  respectively.  This 
increased  activity  can  be  attributed  both  to  the  increased  number  of 
participating  PSROs,  and' increased  PSRO  productivity,  as  displayed 
on  Chart  IV. 
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Distribution  of  Completed  Reaudits  by  Classification 
The  limited  universe  of  completed  reaudits  reduces  the 
reliability  of  any  inferences  drawn  from  the  available  data,  which 
is  displayed  on  Table  XXXVm.  Consequently,  a  discussion  of  the  distri- 
bution of  completed  reaudits  by  classification  can  only  be  indicative 

9 

of  general  tendencies. 

At  this  point  it  seems  that  the  distribution  of  reaudits  in  the 
medical  classification  tends  to  reflect  the  distribution  of  original 
audits,  within  somewhat  broad  limits;  45.2%  of  original  MCE  audits 
addressed  topics  in  Classification  A  (Medical),  as  compared  with  47.7% 
of  the  audits  (including  the  reaudits  submitted  thus  far  for  Quarters'! 
and  2  of  1977). 

Classification  B  (Surgical  received  45.6%  of  initial  MCE 
audit  attention  as  compared  with  37.1%  of  the  reaudits.    The  difference 
between  these  levels  is  too  great  to  be  attributed  to  chance, 
This  8.5%  difference  in  magnitude  suggests  that  when  a  PSRO  or  its 
hospitals  begin  to  perform  reaudits  surgical  topic  categories  receive 
less  attention  than  those  in  the  medical  classification.  Interestingly, 
a  similar  pattern  occurred  with  initial  MCE  audit  activity.  Audits 
of  surgical  topic  categories  were  initially  performed  less  frequently 
than  those  in  the  medical  classification.    However,  over  time  the 
frequency  of  auditing  surgical  topic  categories  increased  to  a  level 
of  parity  with  that  of  medical  topic  categories. 

Finally,  Classification  C,  Administrative  and  Miscellaneous 
Procedures,  was  responsible  for  9.2%  of  original  MCE  audits,  and  15.2% 
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( 


Classifi- 
cation 

A 

B 

C 

TOTAL 


Table  XXXVIII 

Distribution  of  Completed  MCE  Re-Audits  by  Classification 


Period  I 


Number  6f 
re-audi  ts 


8 
6 

14 


Percent  of 
all  re-audits 
in  1975 


57.1 
42.9 

100% 


Period  II 


Number  of 
re-audits 


49 
39 
20 
108 


Percent  of 
all  re-audits 
in  1976 


45.4 
36.1 
18.5 
100% 


1977 

TOTAL 

Percent  of 

Percent 

Number  of 

all 

re-audits 

Number  of 

of 

all 

re-audits 

in 

1977 

re-audits 

re- 

•audits 

6 

60.0 

63 

47.7 

4 

40.0 

49 

37.1 

20 

15.2 

10 

« 

100% 

132 

100% 

of  the  reaudits.    The  difference  between  these  levels  is  too  great 
to  be  attributed  to  chance. 

Thus,  on  the  basis  of  this  limited  data  one  might  infer 
that  the  frequency  distribution  of  reaudits  tends  to  reflect  the 
frequency  distribution  of  orginal  MCE  audits  within  the  medical 
classification  and  to  be  less  than  the  frequency  distribution  of 
initial  audits  in  the  surgical  classification.    Conversely,  the  fre- 
quency of  reaudits  is  higher  than  the  frequency  of  initial  audits 
in  the  administrative  and  miscellaneous  procedures  classification. 

Rank  Ordering  of  the  Most  Frequently  Reaudited  Topic- 
Related  Categories 

As  indicated  on  Table  XXXK  ulcers  and  myocardial  infarction 
were  the  most  frequently  reaudited  MCE  topic  categories  during  the 
period  discussed  in  this  report.    These  two  topic  categories  were 
investigated  11  and  9  times,  respectively.    Pneumonia  has  been  reaudited 
on  "6  occasions,  appendectomy •  and  hip  fracture  both  have  been  reaudited 
5  times,  and  there  have  been  4  reported  reaudits  of  diabetes 
through  the  first  two  quarters  of  1977.    With  the  exception  of  myo- 
cardial infarction,  the  pattern  of  reaudits  varies  somewhat  from 
the  pattern  of  topfc. categories  chosen  for  initial  audits  (see  Table 
IV  -  3.1.7).    Ulcers,  the  first  topic-related  category  in  frequency 
of  reaudit,  was  15th  in  frequency  of  initial  audit.  Appendectomy, 
hip  fractures  and  diabetes  were  ranked  4,  8,  and  8  in  frequency  of 
initial  audit,  and  4.5,  4.5  and  6  in  frequency  of  reaudits. 
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Table  XL 


Comparative  Rank  Ordering  of  the  Twenty  Most  Frequently 
Reaudit  Topic  Categories 


XUpiC  OdlCj^UI  j   ill  UtJfaCt3IlU.lJl^ 

UI  LlC  X    Ul   £  J.  CUUCIly JF 

r ci  luu  i 

Number  of 
re-audits 

Period  II 
Number  of 
re-audits 

1      ?      1077  I 
1  ,    C ,     13//  1 

Number  of 
re-audits 

111  cer 

10 

1 

Myocardial  Infarction 

2 

7 

Pneumonia 

1 

5 

Appendectomy 

5 

Hip  Fracture 

4 

1 

Diabetes 

1 

3 

Cerebrovascular  Accident 

2 

1 

1* 

Choi  ecys tec tony 

1 

3 

Hysterectomy 

c 

3 

1 

Hypertension 

3 

Asthma 

3 

C- Section 

1 

2 

Blood-Transfusions  and  Usage 

3 

Emergency  Room 

3 

Congestive  Heart  Failure 

2 

Hip  Replacement 

2 

Dilatation  and  Curettage 

1 

1 

Laboratories 

Z 

Patients-Admissions,  Discharge,  Transfer 

2 

Antibiotics 

3 

Inguinal  Hernia 

1 

1 

Chronic  Obtrusive  Pulmonary  Disease 

1 

1 

Normal  Pregnancy 

1 

1 

Stapes  Surgery 

1 

Liver  Biopsy 

1 

Post  Operative  Infection 

1 

1 

Cataract 

1 

Anesthesia 

1 

Miscellaneous  (1  study  per  topic) 

40 

TOTAL 

14 

108 

10 
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Conclusions  from  Chapter  5 

The  goal  of  this  chapter  was  to  assess  the  overall  impact 
of  the  MCE  Program.    This  program  can  be  effective  only  if: 

1)  Topic  selection  practices  of  PSROs  enable 
the  identification  of  problem  areas 

2)  Follow-up  practices  of  PSROs  actually  remove 
these  problem  areas. 

To  evaluate  the  potential  of  the  MCE  Program,  therefore,  this  chapter 
first  examines  the  performance  of  the  MCE  Program  in  selecting  MCE 
topics  through  a  system  which  rates  the  commonly  chosen  topic  cate- 
gories by  their  relative  probability  of  containing  MCEs  capable  of 
identifying  problem  areas.    Secondly,  to  evaluate  follow-up,  this 
chapter  measures  the  degree  to  which  PSROs  have  implemented  recom- 
mended actions  as  well  as  the  level  of  reaudit  activity. 


Topic  Selection 

1.  The  medical  diseases  which  are  the  chief  causes 
of  deaths  in  this  country,  except  for  carcinoma, 
are  the  most  frequently  selected  medical  topics. 
Likewise,  the  most  frequently  selected  surgical 
topics  are  the  surgical  procedures  which  are 
most  often  carried  out  (except  those  relating  to 
trauma).    By  choosing  widespread  treatable 
diseases  and  procedures  that  pertain  to  many 
patients,  physicians,  and  ancillary  services, 
the  program  has  a  good  chance  of  revealing 
problem  areas. 

2.  The  actual  procedures  involved  in  topic  selection 
appear  to  be  based  primarily  on  the  desire  of 
PSROs  to  choose  major  acute  diseases. 

a)    The  high  ranking  (top  10)  topic  categories 
which  are  selected  with  high  frequency 
(myocardial  infarction,  appendectomy, 
cholecystectomy,  pneumonia,  hysterectomy) 
are  related  to  acute  hospital  admissions. 
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b)  The  high  ranking  topic  categories  which  are 
selected  with  lower  frequency  are  chronic 
diseases  (hypertension,  cerebrovascular 
accidents,  diabetes). 

c)  The  situation  specific  rationale  for 
selecting  topics  is  frequency  or  "com- 
mon admission"  in  50%  of  the  cases. 
PSROs  presently  seem  to  have  no  other 
generally  used  rationale  for  choosing 
MCE  topics. 

d)  MCE  topics  are  not  usually  focused  on 
a  specific  area  of  their  general  topic 
This  reduces  the  ability  of  the  MCE  to 
reveal  problem  areas.    In  addition, 
because  the  overall  topic  categories 
selected  are  so  important,  it  could  be 
most  useful  for  a  hospital  to  do  several 
MCEs  with  varying  foci  in  these  areas. 
This  can  only  be  done  if  there  is  appro- 
priate focusing  within  topic  category. 

3.      Based  on  1  and  2  above,  the  MCE  Program  is  presently 
identifying  problem  areas.    However,  the  PSRO  prac- 
tices in  topic  selection  will  have  to  be  refined  in 
the  future  to  permit  the  continual  selection  of  high 
impact  topics  and  the  implementation  of  rigorous 
studies. 


Fol low-Up 

1.  Follow-up  is  required  in  more  than  60%  of  MCE  audits. 
Our  data  indicate  that  almost  all  of  these  audits 
make  recommendations  for  remedial  interventions.  It 
appears  that  most  MCEs  implement  these  recommended 
actions,  although  documentation  of  such  implementation 
is  spotty. 

2.  Few  MCE  reaudits  have  been  completed  at  this  point, 
however,  even  though  they  are  necessary.  PSROs, 
therefore,  are  not  getting  feedback  on  efficacy  of 
their  recommended  actions. 

3.  Without  reaudits,  neither  the  Bureau  nor  the  PSROs 
can  easily  evaluate  the  usefulness  of  MCEs.  (In 
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fact,  data  on  recommendations  in  this  section 
are  hard  to  validate  as  a  result.) 

4.     Because  of  1  above  and  because  MCEs  have  been 
shown  to  be  a  worthwhile  working  tool  in  the 
removal  of  deficiencies,  problem  areas,  once 
discovered,  are  probably  being  eliminated  by 
the  PSRO  Program. 

Overall,  the  MCE  Program  appears  to  have  the  potential  to 

have  an  impact  on  medical  care  and  patient  outcomes.    Refinements  in 

the  PSRO  process  of  topic  selection  and  the  consistent  performance  of 

reaudits  will  ensure  that  problem  areas  will  continue  to  be  identified 

and  removed. 
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6.  HOW  HSQB  PLANS  TO  EVALUATE  THE  IMPACT  OF  PSRO 

RETROSPECTIVE  REVIEW  ON  MEDICAL  CARE 

The  Health  Standards  and  Quality  Bureau  of  the  new  Health 
Care  Financing  Administration,  is  responsible  for  further  evaluating 
the  effectiveness  of  the  PSRO  program  in  effecting  changes  in  the 
medical  care  system.    To  date,  scope  of  medical  care  evaluation  activity 
has  focused  on  assessment  of  the  quality  of  care  and  the  efficiency 
of  care  being  delivered  in  acute  care  facilities.    When  the  assessments, 
performed  by  local  physicians  and  other  health  care  practitioners,  have 
revealed  deficient  patterns  of  medical  care  or  patient  outcomes,  local 
physicians  and  other'  health  care  practitioners  have  been  recommending 
and  following  up  the  results  of  actions  made  to  remove  the  deficiencies 
in  care.    As  the  quantity  of  medical  audit  activity  has  increased, 
BQA  and  OPEL  have  been  evaluating  the  effectiveness  of  this  activity. 
The  primary  purpose  of  this  chapter  is  to  outline  future  plans  for 
evluation  of  the  difference  made  in  medical  practice  and  the  quality 
of  care  by  the  implementation  of  MCE  activity  across  the  country. 

This  MCE  Evaluation,  completed  September  20,  1977,  analyzed 
some  of  the  early  changes  brought  about  in  medical  care  and  patient 
outcomes  by  examining  a  sample  of  completed  MCEs  themselves,  consisting 
of  medical  audit- reaudit  combinations  performed  in  the  same  hospital. 
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Variations  from  locally  established  criteria,  defining  the  expected 
practices  in  diagnosing  and  treating  patients,  were  shown  to  be 
identified  in  each  of  the  initial  medical  audits.    After  recommended 
actions  had  been  implemented  in  each  MCE,  examination  of  the  average 
reaudit  revealed  that  68%  of  variations  (in  medical  care  practice 
and  expected  patient  outcomes)  were  being  eliminated.    The  final 
evaluation  report  concluded  that  MCEs  appeared  to  be  making  a  dif- 
ference in  medical  practice  and  patient  outcomes.    However,  the  last 
chapter  of  this  report  also  emphasized  the  limitations  of  this 
evaluation  which  relied  on  reading  MCEs  themselves  to  measure  their 
own  impact. 

One  method  of  externally  assessing  the  impact  of  MCEs  on 
changing  medical  care  process  (including  medical  practice)  is  for  the 
Bureau  to  design  large  scale  medical  audits  using  explicit.medical 
care  process  and  patient  outcomes  criteria.    The  input  of  National 
Council  and  experts  in  the  areas  of  medicine  and  administration 
would  be  essential.    Auditing  would  be  performed  on  a  large  sample 
of  patients  taken  from  specifically  defined  patient  populations  in 
institutions  not  yet  under  binding  PSRO  review.    The  overall  rates 
of  variation  from  each  criterion  would  be  calculated.  Deficiencies 
in  the  way  patients  are  admitted,  given  routine  hospital  care, 
diagnosed,  treated,  and  assigned  appropriate  levels  of  care  would 
be  noted  where  the  variation  rates  were  deemed  inappropriately  high. 
Similar  large  scale  reaudits  would  then  be  performed  two  years  after 
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review  had  been  well  established.    Statistically  significant  reduc- 
tion in  variation  rates  from  medical  process  criteria  would  be  taken 
as  evidence  that  PSRO  retrospective  review  had  made  a  difference  in 
medical  practice.    Unchanged  variation  rates  would  attest  to  the  fact 
that  the  quality  of  care  had  not  been  significantly  improved  by  two 
years  worth  of  PSRO  retrospective  review. 

Another  method  of  assessing  the  impacc  of  MCEs  would  involve 
directly  measuring  the  quality  and  efficiency  of  medical  care  delivered 
before  and  after  the  introduction  of  PSRO  review  into  hospitals. 
Measurable  parameters  for  the  quality  and  efficiency  of  medical  care 
might  include: 

a.  Number  and  types  of  diagnostic  tests  performed 
for  specified  groups  of  patients. 

b.  Number  and  types  of  therapeutic  procedures 
ordered  and  reordered  for  specified  groups 
of  patients. 

c.  Number  and  types  of  hospital  personnel  providing 
services  to  specified  groups  of  patients. 

d.  Patient  questionnaire  scores  on  knowledge  about  their 
diseases. 

e.  Measures  of  physician  and  other  health  care  prac- 
titioner, knowledge. 

f.  Scores  of  patient  satisfaction  with  their  care. 

g.  Provider  admission  rates  for  specified  groups 
of  patients. 

h.  Rates  of  readmission  for  specified  groups  of 
patients. 
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The  parameters  above  would  be  measured  in  a  study  and 
a  control  group  of  hospitals  at  two  points  in  time.    The  first 
group  would  be  measured  before  and  two  years  after  review  is 
established.    The  second  group  of  hospitals,  in  which  no  review 
is  underway  and  in  which  the  prospect  for  establishing  review  is 
unlikely,  would  be  measured  (using  the  above  parameters)  simul- 
taneously.   A  comparison  between  the  aggregate  improvement  in  the 
medical  care  parameters  would  be  made  between  the  two  groups. 

A  similar  methodology  might  be  attempted-  in  assessing  the 
impact  of  PSRO  review  on  the  actual  status  of  patient  health.  Such 
an  evaluation  becomes  extremely  difficult  due  to  the  myriad  of  fac- 
tors which  influence  ultimate  patient  outcomes.    Also,  such  an  eval- 
uation requires  extensive  time  for  the  implementation  of  epidemiologic 
monitors.    However,  an  early  attempt  could  be  made  to  measure  the 
impact  on  patient  health  status  by  assessing  changes  in  one  health 
status  indicator  for  one  selected  disease. 

This  pilot  inquiry  can  be  readily  done  on  the  impact  of 
Myocardial  Infarction  MCEs  on  the  hospital  and  disease  specific 
Mortality  Ratio.    The  epidemiologic  monitor  consists  of  the  PHDDS 
tapes  now  being  completed  which  contain  information  on  each  discharged 
Federal  patient  in  a  PSRO  area.    The  recorded  information  includes 
specific  hospital,  primary  diagnosis,  and  discharge  status  (alive 
or  dead)  for  each  patient.    Using  the  interactive  Autogroup  statistical 
analysis  program,  the  PHDDS  tape  can  be  used  to  calculate  the  mortality 
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rates  for  several  hospitals  which  are  doing  MCEs  in  the  topic  myocar- 
dial infarction.    The  mortality  rates  pre-audit  and  post  reaudit, 
standardized  for  age  and  sex,  would  be  combined  into  a  change  index 
for  each  hospital  in  this  study.    A  hospital  specific  change  index 
would  also  be  calculated  based  on  the  audit  and  reaudit  variation 
rates  noted  from  the  mortality  criterion.    Correlation  co-efficients 
will  be  developed  for  the  two  change  indexes.    A  total  positive 
correlation  between  reduction  in  MCE  variation  rates  and  reduction  in 
mortality  rates  would  be  taken  as  an  indication  that  the  MCE  process 
may  well  effect  patient  health  status.    Such  a  finding  would  motivate 
the  completion  of  further  large  scale  epidemiologic  studies. 
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SECTION  VI 


ROUTINE  MCE  STUDY  ANALYSIS  ASSESSMENT  OF  THE  MCE 
STUDY  PROCESS  AND  ITS  INITIAL  IMPACT  ON 
HEALTH  CARE  PROCESSES 

A.  Introduction 

In  Section  V,  a  procedure  for  establishing  a  modified  reporting  system 
for  MCE  studies  and  a  procedure  for  sampling  based  on  the  reporting  system  in 
order  to  establish  a  data  base  for  further  analysis  was  described  in  some  detail. 
Based  on  the  assumption  that  BQA  will  be  able  to  obtain  from  the  PSROs  the  com 
pleted  study  reports  (BQA  131)  for  those  studies  selected  as  part  of  the  sample, 
the  next  issue  which  must  be  addressed  is  what  is  to  be  done  to  review  and  to  as- 
sess the  sample  reports  which  BQA  will  receive. 

This  section  will  show  purposes  for  routine  assessment  of  the  MCE 
study  process  and  outcome,  the  factors  which  influence  the  performance  of  MCE 
studies,  and  how  these  issues  can  be  dealt  with  by  basic  type  of  review  and 
analysis. 

As  noted  previously,  the  MCE  study  is  the  quality  assurance  com- 
ponent of  the  PSRO  program.   As  such,  it  is  vital  that  BQA  be  able  to  determine 
the  quality  of  the  process  and  the  benefits  of  its  outcomes.   KSI  is  recommending 
that  for  routine  purposes,  a  two  level  system  of  analysis  be  insituted  to  assess 
the  MCE  process  and  a  one  level  system  to  assess  initial  impact  of  the  MCE 
study  process.   The  long  term  impacts,  both  potential  and  actual  will  be  dealt 
with  in  a  later  section  of  this  report. 
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Under  a  two  level  system  of  process  analysis,  the  sample  of  MCE 
studies  will  be  reviewed  at  two  levels  of  depth  and  a  determination  will  be  made 
of  the  adequacy  of  each  study  reviewed  at  either  level.  Judgements  about  adequacy 
will  be  based  on  a  weighted  system  of  critical  components  which  together  consti- 
tute the  total  MCE  process.   Based  on  further  analysis  of  those  studies  rated 
adequate  or  inadequate  further  decisions  can  be  made  about  the  variables  affecting 
the  implementation  of  a  valid  MCE  study  and  both  areas  where  technical  assis- 
tance is  needed  and  areas  where  changes  in  the  MCE  regulations  are  needed  can 
be  identified. 

B.  First  Level  Analysis  of  Sample  MCE  Studies  -  Rating  of  MCE  Studies 

on  Critical  Elements 

The  first  level  analysis  of  the  sample  of  MCE  studies  consists  of  two 
parts.  The  first  is  a  rating  of  "adequate"  or  "inadequate"  given  to  each  of  the 
studies  sampled  and  based  on  a  weighting  of  elements  felt  to  be  essential  to  the 
implementation  of  a  successful  MCE  study.  The  second  part  of  the  analysis 
consists  of  comparisons  and  determinations  made  about  the  factors  which  af- 
fect the  completion  of  a  successful  or  a  non-successful  MCE  study. 

The  quality  of  any  MCE  study  is  a  Function  of  the  following  critical 
elements  of  the  study  process:  reasons  for  topic  selection,  development  of  cri- 
teria and  standards,  determination  of  actual  performance,  comparison  of  actual 
performance  to  standards,  attribution  of  deficiencies,  and  presence  of  plans 
•  for  correcting  deficiencies  if  they  are  found  to  exist.   If  a  certain  number 
types  of  these  elements  are  absent  or  for  some  reason  unacceptable,  the  MCE 
study  should  be  judged  inadequate.   The  basis  for  weighting  the  elements  will  be 
discussed  later  in  this  section.   The  following  section  describes  how  the  charac- 
teristics upon  which  a  judgment  of  acceptability  of  each  of  the  six  study  elements 
is  made. 
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1.  Reasons  for  Topic  Selection:  There  are  three  primary  reasons 
for  selecting  a  topic  for  study.  To  begin  with,  the  condition,  pro- 
cedure, or  administrative  practice  should  occur  relatively  frequently. 
If  topics  of  infrequent  occurrence  are  chosen,  there  will  be  lessened 
opportunity  for  the  study  to  significantly  influence  the  quality  of  the 
health  care  process.  In  the  second  place,  the  probable  relationship 
of  the  topic  to  the  patient's  health  should  be  known.   If  there  is  no 
general  agreement  on  the  potential  effectiveness,  or  benefit  of  the 
topic  selected,  the  study  has  a  high  probability  of  being  non-productive. 
Finally,  the  condition,  procedure,  or  administrative  practice  under 
consideration  should  be  of  high  cost.  Examination  of  costly  activities 
can  indirectly  influence  the  quality  of  care  through  the  reallocation 

of  resources. 

Although  analysis  of  these  three  reasons  for  selecting  a  study  topic 
is  highly  judgmental,  the  topic  selection  has  such  a  substantial  impact 
on  the  overall  quality  of  the  study,  that  a  judgment  should  be  made.  A 
determination  should  be  made  of  the  reason  for  which  the  study  was 
conducted.  If  it  is  apparent  that  the  prevalence,  cost,  or  probable  effect 
on  the  health  of  the  patient  of  the  procedure  or  practice  was  a  reason 
for  conducting  the  study,  then  this  characteristic  should  receive  an 
acceptable  rating. 

2.  Development  of  Criteria  and  Standards:  The  content  of  criteria 
and  standards  is  so  variable  that  basing  an  assessment  of  the  quality 
of  the  study  on  this  characteristic  would  be  very  difficult.  However, 

the  presence  of  written  criteria  and  standards  to  be  compared  with  actual 
performance  is  fundamental  to  the  implementation  of  the  study.  Con- 
sequently, an  MCE  study  should  not  be  considered  adequate  unless  a 
written  list  of  criteria  and  standards  is  submitted  as  part  of  the  study 
report. 
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3.   Determination  of  Actual  Performance:  This  element  of  the  study 
is  composed  of  several  activities.    First,  some  indication  must  be 
given  of  the  number  of  patients  included  in  the  study  and  the  procedure 
by  which  they  were  chosen  from  among  all  similar  patients.   The  cri- 
tical factor  is  that  the  sample  selected  should  be  reasonably  repre- 
sentative of  all  patients  with  the  condition.  Second,  some  data  collection 
instrument  must  have  been  used  which  abstracted  information  corres- 
ponding to  the  written  list  of  criteria  and  standards.   Otherwise,  it  is 
difficult  to  imagine  how  appropriate  comparisons  can  be  made  between 
standards  and  actual  performance.   In  summary,  then,  some  indication 
that  the  sample  of  patients  or  records  are  representative  and  a  copy 
of  the  data  collection  form  are  necessary  to  assess  whether  the  deter- 
mination of  actual  performance  of  the  study  is  acceptable. 

4.  Comparison  of  Actual  Performance  to  Standards:  Although  this 
comparison  has  not  been  produced  in  written  form  in  the  past,  it  is 
a  key  characteristic  of  the  study.   To  state  that  deviations  do  or  do 
not  exist  is  not  specific  enough  to  permit  judgement  of  the  quality  of 
this  characteristic.  There  can  be  substantially  different  implications, 
for" instance,  if  deviations  occur  primarily  among  patient  outcome  stan- 
dards rather  than  process-  standards.  Consequently,  a  written  measure 
of  performance  must  appear  for  each  written  standard. 

5.  Attribution  of  Deficiencies:  Although  it  is  often  difficult  to  estab- 
lish an  explicit  institution,  individual,  or  administrative  practice  re- 
sponsible for  each  deficiency,  it  is  necessary  to  attribute  the  problem 
before  corrective  action  can  be  planned  and  pursued.  A  list  attributing 
deviations  to  someone,  some  group,  or  some  institution  is  necessary' if 
this  study  element  is  to  be  judged  acceptable. 
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6.  Presentation  of  Plans  for  Correcting  Deficiencies:  Finally,  some 
action  must  be  taken  to  correct  any  deficiencies  found.  Again,  while 
it  is  difficult  to  determine  the  content  of  the  actions,  it  is  necessary 
to  determine  whether  they  are  appropriate  to  the  previously  listed 
attributions.   For  example,  an  educational  program  directed  at  phy- 
sicians is  probably  an  inappropriate  action  for  a  deficiency  traced  to 
hospital  administration.   Therefore,  to  judge  this  element,  it  is 
necessary  to  have  a  written  list  of  recommended  actions  which  is 
compatable  with  attributions. 

Table  5  consists  of  the  necessary  components,  or  the  standards  for 
judgment,  for  each  of  the  six  elements  of  an  adequate  MCE  study.   If  any  of 
these  elements  are  not  judged  acceptable,  the  study  is  judged  to  be  inadequate. 

Table  5.   Elements  of  the  MCE  Study  Process  and  Standards 
for  Judging  Their  Acceptability 

1.  Reasons  for  Topic  Selection 

Standard  -  High  prevalence 

-Known  probable  effect  on  patient's  health 
-High  cost 

2.  Development  of  Criteria  and  Standards 
Standard  -  Written  list  of  criteria  and  standards 

3.  Determination  of  Actual  Performance 

Standard  -  Sampling  procedure  which  insures  representativeness 
-Copy  of  data  collection  instrument 

4.  Comparison  of  Actual  Performance  to  Standards 
Standard  -  Written  performance  measures  for  each  standard 
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5.  Attribution  of  Deficiencies 

Standard  -  Written  list  of  "responsible"  organizations,  institutions, 
or  individuals  (generic) 

6.  Presentation  of  Plans  for  Correcting  Deficiencies 

Standard  -  Written  list  of  recommended  actions  compatable  with 
attributions. 


As  noted  before,  KSI  recommends  that  a  weighted  scoring  system  be  used 
to  determine  the  adequacy  of  a  study.  The  reason  for  the  use  of  such  a  system 
is  that  all  six  of  the  study  elements  described  above  do  not  contribute  equally 
to  the  success  of  the  study  process.   For  instance,  experience  and  professional 
jusgment  would  indicate  that  the  reason  for  topic  selection,  and  even  more  so 
the  development  of  criteria  and  standards,  will  effect  the  study  process  to  a 
greater  extent  than  the  sampling  procedure.   KSI  suggests  that  each  study  be 
rated  in  the  following  manner. 

•  A  possible  100  points  is  to  be  allowed  for  each  study. 

•  The  six  elements,  if  judged  acceptable  according  to 
the  standards  indicated  previously,  would  receive  the 
following  point  values: 


1. 

Reasons  for  topic  selection 

20  points 

2. 

Development  of  criteria  and 

standards 

30  points 

3. 

Determination  of  actual 

performance 

10  points 

4. 

Comparison  of  actual  performance 

to  standards 

10  points 

5. 

Attribution  of  deficiencies 

10  points 

6. 

Presentation  of  plans  for 

correcting  deficiencies 

20  points 

100  points 

•     Any  element  judged  not  acceptable  would  receive  no  points 
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•  To  be  given  a  final  rating  of  "adequate"  a  study  would 
have  to  receive  at  least  70  points.   This  allows  for  some 
deviation  from  what  might  be  considered  a  "perfect"  study 
process.   However,  as  time  goes  by,  and  experience 
with  performing  MCEs  is  gained,  this  minimum  standard 
might  be  raised. 

•  A  rating  of  "undetermined"  should  be  given  any  study  in 
the  sample  which  cannot  be  scored  on  the  above  basis. 

In  summary,  for  the  first  level  analysis  of  sample  MCE  studies,  six 
elements  of  the  MCE  process  considered  to  be  critical  to  the  completion  of  an 
adequate  study  have  been  described.   Standards  for  judging  whether  or  not  each 
element  has  been  performed  in  an  acceptable  way  were  suggested.  Finally, 
a  weighted  rating  system  was  suggested  for  the  scoring  of  each  study  considered. 
Once  each  of  the  MCE  studies  in  the  sample  drawn  is  given  a  rating,  more 
detailed  analyses  can  be  performed. 
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DEVELOPMENT  AND  SPECIFICATION  OF  THE  METHOD  OF  APPLICATION  OF 
AN  IMPACT  RATING  SYSTEM  FOR  MCE  STUDIES 

L  INTRODUCTION 

The  requirements  of  Contract  240-77-0074,  An  Immediate  Evaluation 
of  Medical  Care  Evaluation  Studies,  call  for  the  development  and  specification 
of  the  method  of  application  of  an  Impact  Rating  System  for  MCE  Studies.  To 
determine  the  actual  or  potential  impact  of  MCE  study  outcomes  on  medical 
care  practices,  it  is  necessary  to  first  determine  whether  the  hospitals  are  doing 
what  they  are  supposed  to  be  "doing,  i.  e.  performing  MCE's  in  an  acceptable 
manner.  Then  it  is  necessary  to  determine  what  if  any  short  and  long  term  affects 
result  from  the  implementation  of  MCEs  by  the  hospital.  * 

As  a  first  step  in  making  these  assessments,  a  system  is  needed  to 
judge  or  assign  a  score  both  to  the  adequacy  of  the  hospitals'  MCE  study  process 
and  the  short  term  outcomes  (immediate  impact)  of  the  studies  conducted.  This 
grade  should  differentially  indicate  both  how  well  the  studies  were  performed  and 
the  potential  for  their  immediate  impact  upon  health  care  practices.  To  do  this 
the  system  should  permit  identification  and  assessment  of  the  numbers  and  types 
of  deficiencies  identified,  the  number,  types  and  percentages  of  actions  recom- 
mended and  carried  out,  and  finally  the  numbers,  types  and  percentages  of 
deficiencies  corrected. 
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The  sections  that  follow  describe  and  illustrate  the  Impact  Rating  System 
for  MCE  Studies,  as  developed  by  Kappa  Systems,  Inc. ,  including  the  overall 
system  characteristics,  analysis  of  its  composite  individual  factors,  scoring 
and  ranking  of  studies,  and  summary. 
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H.  OVERALL  SYSTEM  CHARACTERISTICS 

Five  factors  were  listed  in  the  original  contract  proposal  for  considera- 
tion in  developing  a  rating  system  for  individual  MCE  studies.  These  were: 

•  An  assessment  of  satisfactory  completion  of  the  elements 
of  the  MCE  study  process  (as  developed  under  Task  I  of 
Contract  HSA-240-75-0015). 

•  The  percentage  of  documented  deficiencies  removed  by 
time  of  restudy. 

•  The  percentage  of  recommended  actions  implemented 
by  time  of  restudy. 

•  The  percentage  of  new  deficiencies  highlighted  by 
restudy. 

•  The  differential  potential  impact  of  given  recommen- 
dations. 

A  rating  system  has  been  devised  as  described  below  which  incorporates 
the  first  three  of  these  factors  as  they  were  originally  conceived.-  The  fourth 
factor  (new  deficiencies)  was  deleted  after  further  consideration  because  it 
was  decided  that  inclusion  would  be  likely  to  distort  the  final  rating's  main 
objective  of  measuring  correction.   This  would  occur  as  there  is  no  current  means 
of  evaluating  the  degree  to  which  new  deficiencies  would  be  corrected  or  the 
degree  to  which  new  deficiencies  were  more  or  less  important  than  the  previously 

identified  deficiencies.  In  addition,  thorough  action  by  audit  committees  in 
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identifying  new  deficiencies  might  be  discouraged  if  such  action  were  assigned  I 
a  negative  value. 

The  fifth  factor  has  been  included  in  the  rating  system  in  a  slightly 
modified  form.  This  factor  is  now  the  differential  potential  impact  of 
corrected  deficiencies  rather  than  of  implemented  recommendations.  This 
factor  value  can  be  used  to  make  comparisons  among  hospitals  in  the  study 
sample  regarding  improvement  of  health  care  within  the  topic  category.  Kappa 
Systems,  Incorporated  does  not  yet  have  sufficient  data  to  apply  this  value  in 
any  but  a  tentative  sense  to  Impact  beyond  the  study  sample  to  the  national 
scene.1  To  validly  attribute  impact  on  a  national  basis,  data  such  as  hospital 
admission  rates  and  community  size  would  be  needed.  This  data  could  be 
obtained  if  sufficient  time  was  available  and  if  the  PSROs  could  be  encouraged 
to  provide  coded  lists  of  their  hospital  discharge  rates.   It  Is  important  to  note, 
however,  that  this  system  was  applied  for  the  purposes  of  this  contract  to  five 
MCE  topic-related  categories  all  of  which  already  have  been  determined  to 
have  substantial  potential  Impact  In  terms  of  cost  of  the  problems,  the  extent 
of  provider  and  patient  populations  involved  and  preventable  morbidity  and 
mortality. 

In  weighting  the  four  factors  ultimately  utilized  in  the  rating  system 
it  was  decided  to  give  equal  weight  to  each:  process;  actions  taken;  deficiencies 
removed;  and  weighted  value  of  deficiencies  removed.  In  essence  this  pro- 
vides twice  as  much  emphasis  for  deficiencies  removed  than  each  of  the  other 
two  factors.  A  system  for  combining  these  values  for  each  complete  MCE 
study  analyzed  has  been  established  on  the  basis  of  a  potential  maximum  of  40 
points.   Twenty-nine  and  one  half  points  was  established  as  the  threshold  of 
acceptability  of  an  individual  study,  based  on  the  decision  to  accept  varying 
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percents  of  scores  as  threshold  for  the  values  contributing  to  the  final  score 
(70%  of  process,  75%  of  actions  taken  and  75%  of  deficiencies  corrected). 
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III.-        ANALYSIS  OF  INDIVIDUAL  FACTORS 
A.    Assessment  of  Process  Factors 

An  assessment  instrument  for  the  first  level  analysis  of  MCE  process 
factors  was  designed  as  shown  in  Figure  l.(page  17).  This  is  based  on  the  sys- 
tem for  first  level  rating  of  critical  MCE  elements  developed  by  Kappa  Systems, 
Incorporated  under  Task  I  of  HSA  Contract  240-75-0015. 

The  criteria  are  designed  to  be  judged  independently  of  one  another 
for  each  study,  and  given  a  maximum  value  of  one  point  each.  In  two  instances, 
topic  selection  and  specification  of  study  design,  two  criteria  are  listed  for  a 
single  unit  value.   For  topic  selection,  "high  impact  category"  (i.  e.  rating 
within  the  first  ten  high  impact  topic  related  categories)  and  "situation  specific" 
are  posed  as  alternate  criteria.   Satisfaction  of  either  earns  the  full  point  value. 
Topic  specificity  is  given  an  additional  point  value  rather  than  contributing  to 
the  topic  selection  score  point  value  as  was  suggested  in  the  previous  deliver- 
able, Development  and  Application  of  a  System  to  Assess  the  Validity  of  the 
Selection  of  Topics  of  MCE  Study,  July  14,  1977.  It  is  felt  that  the  original 
method  is  preferable,  but  the  level  of  topic  specificity  among  the  MCE  studies 
reviewed  was  so  low  that  too  many  studies  would  lose  credits  if  it  counted  as 
part  of  topic  selection  at  this  point  in  time.   In  the  case  of  study  design, 
although  adequacy  of  data  display  was  noted,  satisfaction  of  sampling  procedure 
requirements  was  judged  to  be  sufficient  to  obtain  the  full  unit  value  credit. 
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Half  points  may  be  awarded  when  the  analyst  feels  the  criteria  are  partially,  but 
not  fully  met.  Similarly,  when  criteria  are  met  either  in  the  audit  or  reaudit 
study,  but  not  in  both,  half  points  may  be  assigned  as  long  as  no  more  than  one 
point  is  given  for  each  criterion. 

Provision  is  made  to  note  whether  all  criteria  were  used  in  the 
restudy  or  only  a  subset  of  the  original  audit  criteria.  This  is  for  information 
only  and  does  not  enter  into  the  point  value. 

B.    Percentage  of  Documented  Deficiencies  Removed  by  Time 
of  Restudy 

A  recording  instrument,  the  Audit  and  Reaudit  Analysis  Record, 
was  designed  to  record  deficiencies  removed  by  the  time  of  restudy  as  shown 
in  Figure  2  (page  18).   It  was  felt  that  recording  such  a  summation  is  essentail 
as  it  provides  easy  reference  data  for  later  analysis  of  the  factors  which  lead 
to  high  and  low  ratings  for  individual  studies  or  groups  of  studies  and  for  the 
assessment  of  cumulative  impact  of  similar  deficiencies  across  studies  in  a 
sample. 

Five  sub-factors  were  selected  to  be  recorded  for  each  deficiency 
as  a  basis  for  judgment  regarding  removal  or  non-removal  of  that  deficiency. 

a.  Compliance  standard  selected  by  the  PSRO  for 
that  criterion  (column  3) 

b.  Percentage  of  charts  reviewed  which  met  the 
criterion,  without  variation,  in  the  original 
audit  (column  4) 

c.  Number  of  charts  compared  with  the  sample 
base  (presented  as  a  fraction)  which  were 
judged  "not  justified"  as  variations  in  the 
original  audit  (column  5) 
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d.  Percentage  of  charts  reviewed  which  met  the 
criteria,  without  variation,  in  the  reaudit 
(column  7) 

e.  Number  of  charts  compared  with  the  sample 
base  (presented  as  a  fraction)  which  were 
judged  "not  justified"  as  variations  in  the  re- 
audit  (column  8) 

A  decision  is  then  made,  on  the  basis  of  analysis  of  these  five  fac 
tors,  as  to  whether  or  not  individual  deficiencies  have  been  corrected  by  the 
time  of  reaudit.  The  decision  is  recorded  in  column  9. 

The  following  criteria  are  to  be  used  as  the  basis  for  this  deci- 
sion: 

The  deficiency  is  considered  removed  if  the  unjustified  varia- 
tion or  non-compliance  has  been  reduced  to  less  than  ten  percent 
of  the  study  sample  and/or  diminished  by  at  least  one  half  the 
unjustified  variation  percentage  determined  in  the  original  audit. 
For  example,  reduction  of  a  percentage  from  forty  percent  to 
nineteen  percent,  or  fifty-two  percent  improvement  (leaving  only 
eighty-one  percent  compliance  with  the  standard  but  still  repre- 
senting more  than  a  fifty  percent  improvement)  would  be  considered 
acceptable.  Reduction  of  the  unjustified  variation  from  forty 
percent  to  twenty-one  percent  would  not  be  acceptable  as  it 
represents  only  forty-seven  and  one  half  percent  improvement 
and  more  than  ten  percent  non-compliance  remains.  A  reduction 
in  unjustified  variation  or  non-compliance  from  fifteen  percent  to  • 
nine  percent  would  be  acceptable  because  less  than  ten  percent 
non-compliance  reamins  although  fifty  percent  of  the  variation 
was  not  removed. 
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C.    Percentage  of  Recommended  Actions  Implemented  by 
Time  of  Restudy 

Provision  is  also  made  on  Figure  2  to  record  actions  recommended 
to  correct  each  deficiency  and  to  record  whether  there  is  evidence  in  the  audit 
report  to  show  that  action  had  been  taken  by  the  time  of  restudy.   Initially,  a 
procedure  was  employed  to  note  each  corrective  action  recommended  (sometimes 
more  than  one  per  deficiency)  and  to  grade  the  MCE  on  whether  or  not  there  was 
documentation  that  the  action  had  been  taken.  Two  problems  arose.  The  first 
was  the  some  relatively  minor  deficiencies  were  addressed  by  several  recom- 
mended actions  while  major  deficiencies  might  be  handled  by  only  one.  Also 
the  potential  impact  of  a  recommended  action  might  vary  from  hospital  to  hospital. 
Initially,  an  attempt  was  made  to  classify  corrective  action  recommendations  into 
major  and  minor  actions.  The  project  staff  were  unable  to  arrive  at  a  consensus, 
however,  concerning  realistic  criteria  to  base  such  a  classification,  particularly 
in  view  of  the  paucity  of  hard  data  regarding  implementation  of  recommendations. 
Therefore,  it  was  decided  not  to  assign  differential  impact  potentials  to  action 
recommendations  at  this  time.   Such  assignment  would  be  desirable,  however, 
and  should  be  considered  for  later  use  in  a  more  refined  scoring  system. 
Therefore,  it  was  decided  that  only  one  recommended  action  is  necessary  and 
completion  of  one  such  action  per  deficiency  is  given  full  score.  If,  however, 
no  action  is  recommended  to  correct  a  deficiency,  this  is  regarded  as  an  omission 
and  a  negative  impact  is  assigned.  This  is  done  by  calculating  the  percentage 
of  recommended  actions  implemented  as  a  fraction  using  the  number  of  such 
actions  (one  per  deficiency)  as  the  numerator  and  the  total  number  of  deficiencies 
identified  as  the  denominator. 

The  second  problem  encountered  was  that  documentation  regarding 
the  completion  of  recommended  action  was  not  usually  available  in  the  audit  notes. 
However  in  many  cases,  judging  from  the  removal  of  deficiencies  by  time  of 
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restudy,  it  was  apparent  that  some  action  had  been  taken,  though  not  recorded. 
A  decision  was  reached  that  the  need  to  document  the  implementation  of  action 
in  restudy  notes  had  apparently  not  been  a  clear  requirement  at  the  time  the  MCE 
studies  in  the  sample  were  performed.   Therefore,  liberal  scoring  standards  were 
employed  for  this  sample.  Where  the  intent  to  take  action  is  clearly  listed  in 
the  original  audit  notes  it  is  assumed  to  have  been  taken,  in  the  absence  of  clear 
evidence  to  the  contrary.   It  is  suggested  that  the  hospitals  be  encouraged  to 
document  implementation  of  recommended  actions  and  to  submit  this  documenta- 
tion when  submitting  a  full  study. 

D.   Percentage  of  New  Deficiencies  Highlighted  by  Restudy 

Provision  was  made  initially  to  record  new  deficiencies  on  Figure 
2.   Several  variations  for  scoring  such  new  deficiencies  were  considered. 

•  Positive  score  for  a  new  deficiency  found  in  an  area  where 
compliance  had  been  acceptable  previously  (indicating  addi- 
tional value  from  a  thorough  reaudit) 

•  Negative  score  for  an  additional  deficiency  in  an  area  where 
compliance  had  not  been  acceptable  (indicating  the  recom- 
mended action  had  not  been  effective) 

•  Negative  score  for  a  large  number  of  new  deficiencies  (in 
proportion  to  the  number  found  in  the  original  study)  indica- 
ting the  first  study  had  not  been  thorough,  or  carefully 
done,  or  an  inappropriate  data  base  had  been  employed,  or 
significant  change  had  occurred  between  audits  in  the  com- 
position of  medical  staff  or  in  the  way  health  care  was 
delivered 

After  a  thorough  discussion  among  project  staff  and  the  project 
officer,  the  conclusion  was  reached  that  the  new  deficiency  factor  should  be 
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deleted  from  the  rating  system  for  the  present  time.  Thus,  the  decision  was 
reached  on  the  basis  of  the  following  considerations: 

•  The  major  objective  of  this  portion  of  the  contract  is  to 
assess  the  impact  of  the  follow-up  action  of  the  MCE 
process.  There  is  no  way  to  judge  what  corrective 
action  would  result  from  identification  of  new  deficien- 
cies until  another  restudy  was  performed.  Introduction 
of  a  value  for  these  new  deficiencies  without  analyzing 
the  potential  they  have  to  correct  deficiencies  would 
distort  the  study  score. 

•  The  inclusion  of  a  scoring  factor  which  could  penalize 
PSROs  and  hospitals  for  identifying  new  deficiencies 
might  discourage  the  search  for  such  deficiencies  for 
later  correction.  Such  a  search  is,  of  course,  one  of 
the  main  objective  of  the  MCE  process  at  any  stage  of 
an  audit  or  reaudit. 

E.    Differential  Potential  Impact  of  the  Removal  of  Deficiencies 

As  explained  in  Section  II  of  this  report,  this  factor  was  changed 
from  a  consideration  of  the  impact  of  recommendations  to  that  of  the  differen- 
tial impact  of  the  removal  of  deficiencies. 

Various  scales  for  rating  the  relative  importance  of  deficiencies 
might  be  chosen.   Several,  ranging  up  to  a  weighting  range  of  five  units,  were 
considered.   In  view  of  the  uneven  nature  of  the  data  available  from  the  MCE 
studies  in  the  current  sample,  it  was  decided,  for  the  present,  to  use  only  a 
basic  system  of  classifying  deficiencies  into  major  and  minor  categories.  A 
more  refined  system  should  be  employed  for  future  analyses  when  collection  of 

0 

more  standardized  data  can  be  specified. 

For  the  present  analysis,  deficiencies  were  classified  as  major 

if  they: 

•  specifically  and  directly  affected  patient  health 
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•     though  otherwise  minor,  were  so  great  in  degree  of 
variance  from  the  standard  (more  than  fifty  percent) 
that  in  the  aggregate  they  became  a  major  deficiency 
for  that  hospital. 

Deficiencies  were  classified  as  minor  if  they  were  quite  clearly  attributable  to 
failure  in  hospital  record  keeping  rather  than  a  failure  in  the  clinical  aspects 
medical  process. 

The  classification  of  each  deficiency  was  noted  on  the  Audit  and 
Restudy  Analysis  Record  (Figure  2,  column  6,  page  18) ,  and  the  different 
correction  values  (two  units  for  major  and  one  for  minor)  were  used  in  the 
later  scoring  computation  as  described. 


12 


KAPPA  &  SYSTEMS,  INC. 


IV.         SCORING  AND  RANKING  OF  STUDIES 


For  the  scoring  of  MCE  studies  within  a  given  topic  related  category, 
a  weighting  system  was  selected  apportioning  one  quarter  total  value  to  the  MCE 
process  factor  (maximum  ten  points) ;  one  quarter  value  to  the  implementation  of 
recommended  actions,  (maximum  ten  points);  one  half  value  to  the  correction  of 
documented  deficiencies  by  the  time  of  restudy  (maximum  twenty  points) . 

The  reasoning  behind  this  apportionment  of  weight  was  as  follows: 

•  Careful  and  methodical  construction  and  implementation 
of  the  study  process  is  essential  to  obtaining  valid  results. 
The  topic  must  be  selected  and  defined  with  care  to  investi- 
gate important  current  problems  at  the  hospital,  realistic 
objectives  must  be  identified,  a  representative  sampling 
system  must  be  chosen,  each  case  must  be  investigated 
thoroughly  and  objectively  to  judge  the  appropriateness 

of  care  delivered,  the  results  must  be  recorded  meticul- 
ously, deficiencies  identified  and  clear  action  recommen- 
dations should  be  listed  in  order  to  provide  both  a  basis 
of  judgement  as  to  the  effectiveness  of  current  care  and 
follow-up  actions  to  correct  deficiencies  which  are  found. 
Without  a  carefully  executed  initial  audit  to  focus  attention 
on  need  for  future  action,  deficiencies  can  be  overlooked  and 
it  may  be  difficult  to  determine  appropriate  actions.  Therefore, 
project  staff  felt  that  an  important  weight  must  be  given 
to  the  initial  study  process.  Ten  points  were  selected. 

•  The  second  element  in  the  grading  system,  degree  to  which 
recommended  actions  were  taken,  is  an  obvious  element  in  the 
follow-up  process.  Project  staff  felt  that  implementation 

of  recommended  actions  should  be  automatic  in  almost 
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every  case.    Nevertheless,  it  was  felt  that  credit  should 
be  given  for  methodical  implementation  of  these  actions. 
On  this  basis,  a  second  ten  points  was  apportioned  to  this 
action  follow-up  category,  making  it  equal  to  the  process 
elements  section.   Documentation  of  implementation  of 
actions  was  rarely  available,  so  for  the  sample  to  which 
this  system  was  applied  it  was  assumed  that  actions  had 
been  taken  and  ten  points  were  given  for  every  study.  This 
automatic  award  of  ten  points  may  prove  to  be  excessive, 
at  least  until  more  data  is  available  including  more  pre- 
cise documentation  as  to  which  actions  were  actually  taken, 
which  in  turn  allow  a  categorization  into  major  and  minor 
actions. 

•  Project  staff  felt  that  greatest  weight  must  be  given  to  the 
correction  of  deficiencies.  Twenty  points,  equal  to  the 
sum  of  the  two  other  categories,  were  reserved  for  this 
element.  Two  considerations  apply  to  deficiencies;  the 
absolute  number,  or  proportion  of  those  corrected,  and 
the  relative  importance  of  those  corrected.  Ten  points 
were  given  to  each.  We  feel  this  is  appropriate  at  this 
stage  of  development  of  the  impact  rating  system  when 
the  scoring  applies  to  the  impact  of  the  restudy  process 
on  medical  care  within  a  given  hospital.  At  a  later  time 
when  more  completed  studies  are  available  as  a  study 
base  and  more  data  per  study  is  available,  such  as  hospi- 
tal bed  size,  discharge  numbers  and  number  of  cases  of 

a  given  disease,  the  scoring  system  may  be  applied  to 
estimate  impact  of  a  study  on  given  communities  and 
eventually  on  a  national  basis.  At  that  time,  KSI  recom- 
mends a  higher  point  value  (and  therefore  higher  impact 
differential)  for  the  last  item  scored  —  the  relative 
importance  (major  or  minor)  of  the  deficiencies  corrected. 

The  mechanical  procedure  for  computing  the  impact  grade  is  as 

follows: 

•  The  grade  for  the  first  element  is  computed  on  the  First 
Level  Analysis  sheet  (Figure  1,  page  17  )  and  the  total 
(ranging  from  0  to  10  units)  transferred  directly  to  the 
MCE  Second  Level  Analysis  sheet  (Figure  3,  page  19). 
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•  The  grade  for  the  second  element  (percentage  recommended 
action  implemented)  is  arrived  at  by  a  straight  forward 
conversion  of  percentage  of  actions  implemented  to  a  scale 
of  0  to  10.  This  is  done  by  multiplying  that  percentage  by 

a  0. 1  conversion  factor.  It  is  then  entered  on  Figure  3, 
(page  19). 

#  Computation  of  the  value  for  correction  of  documented 
deficiencies  is  slightly  more  complicated  in  that  it  com- 
bines the  percentage  of  deficiencies  corrected  with  a 
separate  computation  for  the  weighted  percentage  of 
deficiencies  corrected.   This  is  shown  on  the  Second 
Level  Analysis  Worksheet  (Figure  4,  page  20).  Percent- 
age of  deficiencies  corrected  is  a  straight  forward  ratio 
of  the  number  of  deficiencies  corrected  compared  to  the 
total  number  identified  in  the  original  audit.   The  weighted 
value  of  deficiencies  corrected  is  obtained  by  assigning 

a  value  of  two  points  each  to  major  deficiencies  and  one 
point  each  to  minor  ones.   The  value  of  the  deficiencies 
corrected  is  totaled  and  taken  as  a  percentage  of  the  total 
potential  value  of  all  deficiencies  which  were  identified. 
The  two  percentages  are  then  averaged  to  obtain  an  aver- 
age deficiency  correction  value.   In  order  to  arrive  at  the 
correct  weight  for  this  third  element  of  twice  the  value  of 
the  other  two,  a  conversion  factor  of  0. 2  is  used,  providing 
a  point  range  of  0-20.   The  value  is  entered  on  the  MCE 
Second  Level  Analysis  sheet,  (Figure  3,  page  19). 
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V.  SUMMARY 

t 

An  impact  rating  system  has  been  developed  to  provide  a  mechanism  for 
reviewing  completed  MCE  studies  in  order  to  determine  the  combined  adequacy 
of  the  study  process  and  the  potential  impact  of  the  study  results.   Both  the 
development  and  method  of  application  of  the  rating  system  were  described. 
This  system  will  be  applied  to  approximately  fifty  studies  selected  from  those 
available  for  five  topic  related  categories  (myocardial  infarction,  pneumonia, 
cholecystectomy,  appendectomy  and  ulcer)  so  that  statements  can  be  made  about  I 
the  quality  of  MCE  study  process  going  on  nationally  and  about  the  potential  impact 
which  the  changes  relating  to  medical  care  process,  which  were  observed  in  the 
sample,  can  have  on  patient  outcomes.  j 
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APPENDIX  C 
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Study  Characteristics 
1.    Topic  Related  Category 


2.    Audit  Title 


3.    Re-audit  Title 


4.    Appropriate  Topic  Specificity 


5.  Method  for  Selecting  Study 

  Profile  Analysis 

''   Concurrent  Review 

  Other  MCE  Study 

  Analysis  of  Medical  Records 

  Perceived  Need 

  Other  (Specify) 

6.  Topic  Characteristics 

  Incidence/Prevalence 

  Preventable  Morbidity/Mortality 

_  Cost  of  Care 
  Other  (Specify) 

7.  Reasons  for  Topic  Selection 

  High  Potential  Impact 

  Situation  Specific  Factors 


8.    Person  Hours  Utilized 


AUDIT 

RE-AUDIT 

no.  ol 

total  num.  of 

no.  oL 

total  num.  oi 

type: 

persons : 

person  hours 

type: 

persons: 

person  hours 

physician 

physician 

oilier  (specify) 

other  (specify) 

9.  Locus  of  Hospital  Study  Responsibility 

Medical  Audit  Committee 
UR  Committee 
Service/Dept.  (Specify) 

Other  (Specify) 

10.  Derivation  of  Criteria 

Self  Generated 

National  Organization 

  Other  PSRO(S) 

  Other  Hospitals  (s) 

  Other  (Specify) 

11.  Written  List  of  Criteria 


12.  Number  of  Criteria  Elements 

13.  All/Subset  criteria  used  for  re-audit 

14.  Written  List  of  Standards 


15.  Study  Site 

  PSRO-Widc 

  PSRO  SUB-AREA  (No.  Hospitals) 

Individual  Hospital 
  Service/Dept.  of  Hosp.  (Specify) 

  Other  (Specify) 

16.  Specification  of  Study  Design 

  Sampling  procedure  to  ensure  representativeness 

Data  Collection  Instrument: 

  Special  Form 

JCAH/PEP 
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CMA 

Routine  Hospital  Form 
Other  •  ■ 


17.  Type  of  Data  Collection 

Retrospective 

Concurrent 

Mixed 


18.  Data  Display 


19.  Written  Tally  Percentage  Comparing 
Actual  Performance  to  Standards 

AUDIT 

20.  Study     Sample  Population 

•  Age: 
Sex: 
Size: 


Study- 
Age: 
Sex: 
Size: 


RE -AUDIT 
Sample  Population 


21 .  Time  Period  for  Sample  Selection 

Audit: 
Re-audit: 

22.  Number  of  Physicians  Audited:  Audit: 
23    Other  Sampling  Characteristics 


Rc— audit: 


24.  Person  hours  utilized:  Physician 

Other 


25.    Available  information  Sources 
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MCE  STUDY  IMPACT  GHADB 


Process  Grade  . 
(1st  level  rating) 


Impact  Grade 
(2nd  Jcvcl  rating) 


Acceptability 


c 


